Unable to read from SharePoint Lists since export of 37K+ items

As discussed the .net network tracing needs to be enabled on PNAP501 and then a Full import run on the SharePoint connector in IdB. Once captured the network trace and IdB logs needs to be sent to Adam for analysis.

Can you not match the existing .NET traces with an IdB error log? Or did it not throw the same error this time around? If the latter is true, then what was done differently such that an error wasn't thrown?

Changes implemented, Full Imports run on External Users, External Roles, and then School Roles. School Roles still running. Will complete in several hours. Logs too large to upload to Jira already.

Adam, It did not throw the same error as I ran the trace on the Dev server because I want to test that the trace was going to pull the information needed.

Matt, I believe that you only need to run an import on the External User connector in IdB then you should see the error in the logs that Boyd detailed.

Matthew, could you please also upload the Unify logs? If you could point out the time that the error occurred that would also be very helpful

Thanks.

Its in the Zip - UnifyLog.txt.txt

Ah so it is. Thanks.

Adam, do you need the logs for the SharePoint Roles Connector? Its at 1.2GB at the moment.

Not if you had the error occur for the stuff that I already have.

OK, let me know. Ill leave it running for the time being. Its up to 1.5GB now, and only processed about 10% of the pages. I've got 20GB spare on the disk so it probably wont fill up, but I am not sure how much use a 15GB log file will be.

I have gone through the logs and can't see anything sticking out. I have tried to parse the results from the trace and there may be an issue with the line-feed character (hex: 0A or appears like ..).

Would you be able to try two separate lines of thought:

1. Trace a list request that works. I'll compare the response with the non-working one.
2. Create a view of the failing list, that filters out everything. Ensure that the connector works against this view and returns nothing. If:
   1. It fails - let me know. This means that it's not a particular item that is failing.
   2. It succeeds - progressively loosen the filter so that more and more items are let through the view. If a particular item is failing the xml, then we should be able to fix up that item.

Please let me know if that doesn't make any sense. Thanks.

Adam: I'd like to propose a Lync meeting tomorrow morning so that you can show me what you're talking about. Ill be onsite, and can share my screen & allow you to drive.

Is there a particular part of my previous comment that doesn't make sense?

For the first statement, it's the same trace you have done previously, but for a list that isn't showing the same error.

For the second statement, I won't be much help as I have never created a view for a SharePoint list. As far as the configuration in IdB is concerned, the setting is there in xml and can be set just like the list name. For details on how to set up a view, http://office.microsoft.com/en-us/windows-sharepoint-services-help/working-with-sharepoint-lists-part-2-HA001129865.aspx?CTT=5&origin=HA001119988 may be of assistance.

If this still does not help, create the meeting and I will be happy to clarify anything.

I didnt understand what you meant by 'list request'.

Under 100 line items works without issue.
Over 100 line items, it appears to detect the incorrect number or items.

Logs uploaded.

I don't see the XmlException in trace, did it not fail this time?

The rowLimit is for paging through items, as per http://msdn.microsoft.com/en-us/library/lists.lists.getlistitems.aspx.

I did notice a comment on an another article http://social.msdn.microsoft.com/Forums/hi-IN/sharepointdevelopment/thread/b6013abd-f659-44fa-963c-bb9fe4bf89b6, where the incorrect number of items were coming through, and was resolved by changing a setting on the default view for the list in SharePoint, as well as a blog post on http://ucuuba.wordpress.com/2011/04/07/sharepoint-list-item-limit-foils-me-again/. I can't comment on whether this would work, or even makes sense, but it's certainly a place to look.

I asked Boyd for an import that works and an import that doesnt so you could compare as per first statement. Not sure if its any help as its a different connector(SharePoint School Roles Connector). The SharePoint School Roles Connector appears to go into some type of loop if there are more than 100 items (including header) in the list. Boyd thought it might be related, but it appears it is not. Ill get him to open another issue if we can't get it working. I dont think changing the row limit is a good fix.

Ah I see. Yes, if you could it would make it easier to keep everything separate unless it's determined that they are related.

I removed all but 5 users. still fails. I am in the office tomorrow and will show you. There does not appear much more I can do to make it simpler.

Emptied List completely. Issue still occurs.

Fantastic, thanks.

I had another look through the code and the stack trace, and came up with another theory. Please see https://unifysolutions.jira.com/wiki/pages/viewpage.action?pageId=18088091.

Download v3.0.5.7 DEV version of Identity Broker for Microsoft SharePoint. This should tell us exactly where the problem is occuring.

Thanks.

please provide link for download, I cant find in the usual places:
https://unifysolutions.jira.com/wiki/display/IDBSP/Downloads
https://unifysolutions.jira.com/wiki/display/BCE/Downloads
https://unifysolutions.jira.com/wiki/display/SUBIDB/Downloads

please reassign to paul when complete as I am on leave mon/tues.

Paul, the download mentioned in the above few comments can be found on SUBIDBSP:Downloads.

Please ensure that you back up the Services directory prior to uninstalling Identity Broker for SharePoint, and then installing this newest version. Keep in mind this is a development version, that is based off the relatively stable candidate, with the addition of some logging. There is a good chance that this will expose the nature of the fix that needs to be made.

Please let me know if you have any problems.

Thanks.

Services folder has been backed up and IdB SharePoint v3.0.5.7 is installed.

Hi Paul,

I think you have referenced both BCE-187 and BCE-188 in the above comment. I will comment on BCE-187 on that issue.

Regarding this issue, the only difference that you should see is additional logging when the xml exception is thrown. Please let me know what you find.

Thanks.

Comment deleted, edited and moved to BCE-187

Hi Adam,

IdB logs have been attached. I have looked at them and they seem to contain extra information that you are looking for.

Hi Paul,

I have attached Unify.Communicators.Moss2007List.dll. This file will work with the DEV install that you used to diagnose this issue with. Once you have confirmed it, then I will aim to get a proper release of this version out.

Thanks.

Adam, I moved the updated dll to the services folder, started the IdB service and then ran a full import on the SharePoint External users connector. Looked at the log and the same error is still being produced.

Attached the logs for you to view.

Sorry about that, a .NET component wasn't working as advertised.

I have attached a new version of Unify.Communicators.Moss2007List.dll using a component that I have written myself to encode the value correctly.

Hi Boyd,

Deploying the new dll seems to have fixed the error, so I think we should run with this and monitor until the changes made are release as part of the next build.

Thanks Paul
I can see it is importing from SharePoint at the moment. I will let it finish and then start testing.

Boyd

From: Paul Barratt (JIRA) jira@unifysolutions.jira.com
Sent: Tuesday, 5 June 2012 1:54 PM
To: Boyd Bostock
Subject: JIRA (BCE-188) Unable to read from SharePoint Lists since export of 37K+ items

https://unifysolutions.jira.com/s/en_US-w35fje/732/89/_/jira-logo-scaled.png

https://unifysolutions.jira.com/secure/useravatar?ownerId=paul.barratt&avatarId=10831Paul Barratt<https://unifysolutions.jira.com/secure/ViewProfile.jspa?name=paul.barratt> assigned Client Support BCE-188<https://unifysolutions.jira.com/browse/BCE-188> to Boyd Bostock (BCE)<https://unifysolutions.jira.com/secure/ViewProfile.jspa?name=bce.boyd.bostock>
Unable to read from SharePoint Lists since export of 37K+ items<https://unifysolutions.jira.com/browse/BCE-188>

Hi Boyd,

Deploying the new dll seems to have fixed the error, so I think we should run with this and monitor until the changes made are release as part of the next build.

Change By:

Paul Barratt<https://unifysolutions.jira.com/secure/ViewProfile.jspa?name=paul.barratt> (05/Jun/12 1:52 PM)

Assignee:

Paul Barratt Boyd Bostock (BCE)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators<https://unifysolutions.jira.com/secure/ContactAdministrators!default.jspa>.
For more information on JIRA, see: http://www.atlassian.com/software/jira

– Disclaimer: The content of this e-mail is the view of the sender or stated author and does not necessarily reflect the view of Brisbane Catholic Education. The content, including attachments, is a confidential communication between the sender or stated author and the intended recipient. If you are not the intended recipient, any use, interference with, disclosure or copying of this e-mail, including attachments, is unauthorised and expressly prohibited. If you have received this e-mail in error please contact the sender immediately and delete the e-mail and any attachments from your system.
Brisbane Catholic Education does not warrant or represent that this email is free from viruses, malicious software or defects.

Hi Paul
I have noticed that the timings on the SharePoint connectors are a Full Import every 10 minutes. This will never finish.

Could you please run the following on all 3 SharePoint connectors

• Full Import at 4PM

• Sync Import every 10 minutes.
  The eMinerva ones also need to be scheduled

• Full 5PM

• Sync every 30 minutes
  Boyd

From: Paul Barratt (JIRA) jira@unifysolutions.jira.com
Sent: Tuesday, 5 June 2012 1:54 PM
To: Boyd Bostock
Subject: JIRA (BCE-188) Unable to read from SharePoint Lists since export of 37K+ items

https://unifysolutions.jira.com/s/en_US-w35fje/732/89/_/jira-logo-scaled.png

https://unifysolutions.jira.com/secure/useravatar?ownerId=paul.barratt&avatarId=10831Paul Barratt<https://unifysolutions.jira.com/secure/ViewProfile.jspa?name=paul.barratt> assigned Client Support BCE-188<https://unifysolutions.jira.com/browse/BCE-188> to Boyd Bostock (BCE)<https://unifysolutions.jira.com/secure/ViewProfile.jspa?name=bce.boyd.bostock>
Unable to read from SharePoint Lists since export of 37K+ items<https://unifysolutions.jira.com/browse/BCE-188>

Hi Boyd,

Deploying the new dll seems to have fixed the error, so I think we should run with this and monitor until the changes made are release as part of the next build.

Change By:

Paul Barratt<https://unifysolutions.jira.com/secure/ViewProfile.jspa?name=paul.barratt> (05/Jun/12 1:52 PM)

Assignee:

Paul Barratt Boyd Bostock (BCE)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators<https://unifysolutions.jira.com/secure/ContactAdministrators!default.jspa>.
For more information on JIRA, see: http://www.atlassian.com/software/jira

– Disclaimer: The content of this e-mail is the view of the sender or stated author and does not necessarily reflect the view of Brisbane Catholic Education. The content, including attachments, is a confidential communication between the sender or stated author and the intended recipient. If you are not the intended recipient, any use, interference with, disclosure or copying of this e-mail, including attachments, is unauthorised and expressly prohibited. If you have received this e-mail in error please contact the sender immediately and delete the e-mail and any attachments from your system.
Brisbane Catholic Education does not warrant or represent that this email is free from viruses, malicious software or defects.

All three SharePoint connectors updated in the ConnectorEngine.extensibility.config

        <getAllEntities>
          <timing name="Daily" offset="06:00:00"/>
        </getAllEntities>
        <polling>
          <timing name="RecurringTimespanStandardTime">
            <timespan value="00:10:00" />
          </timing>
        </polling>

And both eMivera connectors

        <getAllEntities>
          <timing name="Daily" offset="07:00:00"/>
        </getAllEntities>
        <polling>
          <timing name="RecurringTimespanStandardTime">
            <timespan value="00:30:00" />
          </timing>
        </polling>

NOTE: Times are in UTC format

Hi Paul
The Synchronization imports on the SharePoint connectors seem to be failing. Does versioning need to be enabled or other Lists settings need to be changed for the Sync Import to work?

Boyd

From: Paul Barratt (JIRA) jira@unifysolutions.jira.com
Sent: Tuesday, 5 June 2012 3:46 PM
To: Boyd Bostock
Subject: JIRA (BCE-188) Unable to read from SharePoint Lists since export of 37K+ items

https://unifysolutions.jira.com/s/en_US-w35fje/732/89/_/jira-logo-scaled.png

https://unifysolutions.jira.com/secure/useravatar?ownerId=paul.barratt&avatarId=10831Paul Barratt<https://unifysolutions.jira.com/secure/ViewProfile.jspa?name=paul.barratt> commented on Client Support BCE-188<https://unifysolutions.jira.com/browse/BCE-188>
Unable to read from SharePoint Lists since export of 37K+ items<https://unifysolutions.jira.com/browse/BCE-188>

All three SharePoint connectors updated in the ConnectorEngine.extensibility.config

<getAllEntities>

<timing name="Daily" offset="06:00:00"/>

</getAllEntities>

<polling>

<timing name="RecurringTimespanStandardTime">

<timespan value="00:10:00" />

</timing>

</polling>

And both eMivera connectors

<getAllEntities>

<timing name="Daily" offset="07:00:00"/>

</getAllEntities>

<polling>

<timing name="RecurringTimespanStandardTime">

<timespan value="00:30:00" />

</timing>

</polling>

NOTE: Times are in UTC format

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators<https://unifysolutions.jira.com/secure/ContactAdministrators!default.jspa>.
For more information on JIRA, see: http://www.atlassian.com/software/jira

– Disclaimer: The content of this e-mail is the view of the sender or stated author and does not necessarily reflect the view of Brisbane Catholic Education. The content, including attachments, is a confidential communication between the sender or stated author and the intended recipient. If you are not the intended recipient, any use, interference with, disclosure or copying of this e-mail, including attachments, is unauthorised and expressly prohibited. If you have received this e-mail in error please contact the sender immediately and delete the e-mail and any attachments from your system.
Brisbane Catholic Education does not warrant or represent that this email is free from viruses, malicious software or defects.

Is the target system Microsoft SharePoint 2007 or 2010?

As per IDBSP305:Microsoft SharePoint List Connector, synchronisation imports are only supported by Microsoft SharePoint 2010.

Hi Adam
I have not been able to open the link provided to read the article (Page Not Found). The list is on SharePoint 2010.

Boyd

From: Adam van Vliet (JIRA) jira@unifysolutions.jira.com
Sent: Wednesday, 6 June 2012 9:14 AM
To: Boyd Bostock
Subject: JIRA (BCE-188) Unable to read from SharePoint Lists since export of 37K+ items

https://unifysolutions.jira.com/s/en_US-w35fje/732/89/_/jira-logo-scaled.png

https://unifysolutions.jira.com/secure/useravatar?ownerId=adam.vanvliet&avatarId=10633Adam van Vliet<https://unifysolutions.jira.com/secure/ViewProfile.jspa?name=adam.vanvliet> commented on Client Support BCE-188<https://unifysolutions.jira.com/browse/BCE-188>
Unable to read from SharePoint Lists since export of 37K+ items<https://unifysolutions.jira.com/browse/BCE-188>

It is target system Microsoft SharePoint 2007 or 2010?

As per IDBSP305:Microsoft SharePoint List Connector<https://unifysolutions.jira.com/wiki/display/IDBSP305/Microsoft+SharePoint+List+Connector>, synchronisation imports are only supported by Microsoft SharePoint 2010.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators<https://unifysolutions.jira.com/secure/ContactAdministrators!default.jspa>.
For more information on JIRA, see: http://www.atlassian.com/software/jira

– Disclaimer: The content of this e-mail is the view of the sender or stated author and does not necessarily reflect the view of Brisbane Catholic Education. The content, including attachments, is a confidential communication between the sender or stated author and the intended recipient. If you are not the intended recipient, any use, interference with, disclosure or copying of this e-mail, including attachments, is unauthorised and expressly prohibited. If you have received this e-mail in error please contact the sender immediately and delete the e-mail and any attachments from your system.
Brisbane Catholic Education does not warrant or represent that this email is free from viruses, malicious software or defects.

Boyd,

I will arrange access to that space.

Paul,

Are you able to collect logs and other relevant information for the above mentioned issue?

Thanks.

Hi Boyd,

Looks like duplicates are causing the sync imports to fail. Here is a error from the IdB logs

Timestamp Source Module Message
6/06/2012 8:44:39 AM Save entities to connector failed. Connector "Save entities Count:1 to connector SharePoint External Users Connector failed with reason Error returned from SharePoint when attempting to modify the list:
Error Code: 0x810200bf
Error Text: The list item could not be added or updated because duplicate values were found in one or more fields in the list.. Duration: 00:00:00.0937494
Error details:
System.Exception: Error returned from SharePoint when attempting to modify the list:
Error Code: 0x810200bf
Error Text: The list item could not be added or updated because duplicate values were found in one or more fields in the list.
at Unify.Communicators.Moss2007ListWebServiceCommunicatorBase.ModifyListItemAndReturnId(IEnumerable`1 data)
at Unify.Connectors.MossListConnector.SaveEntities(IEnumerable`1 entities)
at Unify.Framework.ConnectorToWritingConnectorBridge.SaveEntities(IEnumerable`1 entities)
at Unify.Framework.EventNotifierWritingConnectorDecorator.SaveEntities(IEnumerable`1 entities)
at Unify.Framework.Adapter.SaveEntities(IEnumerable`1 entities, Boolean reflect)
at Unify.Framework.Adapter.SaveEntity(IAdapterEntity entity, Boolean reflect)
at Unify.Framework.CompositeAdapter.SaveEntity(IAdapterEntity entity)
at Unify.Framework.AdapterNotifierDecorator.SaveEntity(IAdapterEntity entityToSave)
at Unify.Framework.LDIFAdapter.ExportAdapterEntity(IAdapterEntity adapterEntity, Guid adapterId)
at Unify.Framework.LDIFAdapterServiceHostDecorator.ExportAdapterEntity(IAdapterEntity adapterEntity, Guid adapterId)
at SyncInvokeExportAdapterEntity(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)"

Timestamp Source Module Message
6/06/2012 8:44:39 AM Adapter request to save entity to adapter space failed. Adapter "Adapter request to save entity 916092af-848a-479b-b12d-c181c066cd09 to adapter space 85c3f1c8-970a-11e0-8262-b8ee4724019b failed with reason Error returned from SharePoint when attempting to modify the list:
Error Code: 0x810200bf
Error Text: The list item could not be added or updated because duplicate values were found in one or more fields in the list.. Duration: 00:00:00.2656233
Error details:
System.Exception: Error returned from SharePoint when attempting to modify the list:
Error Code: 0x810200bf
Error Text: The list item could not be added or updated because duplicate values were found in one or more fields in the list.
at Unify.Communicators.Moss2007ListWebServiceCommunicatorBase.ModifyListItemAndReturnId(IEnumerable`1 data)
at Unify.Connectors.MossListConnector.SaveEntities(IEnumerable`1 entities)
at Unify.Framework.ConnectorToWritingConnectorBridge.SaveEntities(IEnumerable`1 entities)
at Unify.Framework.EventNotifierWritingConnectorDecorator.SaveEntities(IEnumerable`1 entities)
at Unify.Framework.Adapter.SaveEntities(IEnumerable`1 entities, Boolean reflect)
at Unify.Framework.Adapter.SaveEntity(IAdapterEntity entity, Boolean reflect)
at Unify.Framework.CompositeAdapter.SaveEntity(IAdapterEntity entity)
at Unify.Framework.AdapterNotifierDecorator.SaveEntity(IAdapterEntity entityToSave)
at Unify.Framework.LDIFAdapter.ExportAdapterEntity(IAdapterEntity adapterEntity, Guid adapterId)
at Unify.Framework.LDIFAdapterServiceHostDecorator.ExportAdapterEntity(IAdapterEntity adapterEntity, Guid adapterId)
at SyncInvokeExportAdapterEntity(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)"

Can you check the data to see if you can weed out any duplicates?

Attached the logs incase there isn't enough info in my last comment.

Hi Paul,

I suspect the Modified field is missing from the configuration.

eg

<field name="Modified" validator="timestamp" readonly="True" />

Hi Adam,

I have added the Modfied attribute to the entity schema and there are no errors, but I cannot see any polling activity in the logs. Is this expected?

Log file attached. The polling import on SharePoint External Users Connector at 00:56:11 UTC was started manually.

Are you saying that the import started, but does not complete? As this log would indicate. Does Management Studio appear to still be running for that connector?

Yes the manual polling import started but doesn't seem to finish and Management Studio doesn't show any activity on that connector. Also, there two other SharePoint connectors that have polling imports scheduled every 10 mins and there is no polling import activity on any of these connectors that I would expect.

In summary:

• Manually started polling import doesn't seem to finish
• Scheduled polling imports on any of the SharePoint connectors doesn't seem to be occurring before or after the manual polling import on the SharePoint External Users connector was initiated.

Could you please raise a new issue for the delta issue?

I would like to resolve this one if possible.

Thanks.

Opened BCE-189 for the scheduled polling import issue

Updated billing key.

Resolved. Pending release for confirmation.

Confirmed working. Issue closed.