1. UNIFYBroker Forum
  2. UNIFYBroker/Microsoft SharePoint
  3. Bugs
0
Fixed

Write-back for SharePoint List Connector fails for nulls

Bob Bradley 9 years ago in UNIFYBroker/Microsoft SharePoint • updated by anonymous 8 years ago • 38

Experiencing issues clearing out the start/end dates defined in the schema of my WSS list. FIM export flow rule is set to "allow nulls" which means any existing values in the WSS list should be cleared - but there is no change and I get the dreaded "exported-not-reimported" error. Have confirmed that WSS list fields allow nulls.


Network Trace - No Patch.log
Network Trace - Patch Applied.log
Network Trace - Soap Exception.log
Unify.Connectors.Microsoft.SharePoint.zip
Unify.Product.IdentityBroker.AdapterEnginePlugInKey.extensibility.config.xml
Unify.Product.IdentityBroker.ConnectorEnginePlugInKey.extensibility.config.xml
Bob Bradley 9 years ago

Hi Adam van Vliet - this only became apparent with a different set of test data, and the need to clear out values. Not dead urgent, as most records should have a start date, but it is expected that lots will have an open (null) end date, or that end dates can change/be cleared.

Adam van Vliet 9 years ago

Hi Bob Bradley,

Have you performed a trace to see what Identity Broker is sending out? A potential cause could be that a few releases of Identity Broker v4.1.x there were issues removing previous values (see BCE-356 for details).

Adam van Vliet 9 years ago

Hi Bob Bradley, were you able to check up on this issue?

Bob Bradley 9 years ago

Adam van Vliet - not yet, but we're in the middle of a large SIT deployment and if there is any issue it is sure to come up here.

Bob Bradley 9 years ago

Adam van Vliet - I am seeing this again now in both DEV and Production. I am too stretched right now to deal with this one issue alone, but I will need to come back to this very soon as it will become a blocker. If there's anything you already know about this problem for which a patch may already be available, I'm willing to try it first because I am way out of troubleshooting time .

Bob Bradley 9 years ago

Richard Green - perhaps if you have some bandwidth you may be able to help me diagnose this issue in DEV today - with Adam van Vliet if he happens to be able to spare time to assist?

In DEV the SARa MA is in an infinite loop for 2 users where the attempt by FIM to delete the end date property is failing in the connector (but not reporting an error back to FIM).

Thanks.

Adam van Vliet 9 years ago

Thanks Bob Bradley. Richard Green, perhaps try with Unify.IdentityBroker.Adapter.LDIF.dll from BCE-356. If it still fails then you may need to trace the call being made to SharePoint - IDBSP41:Enabling Network logging (there's a chance it's still configured, but commented out by Bob).

Richard Green 9 years ago

Hi Adam,

Looks like it still has the same issue with the patch from BCE-356 in place. Before I rule this out though, I tested it with the patch in the services directory as indicated on the link, however I notice there is also a patches directory in the services folder. Is there a difference to files placed in this folder as opposed to the base services directory?

Adam van Vliet 9 years ago

Hi Richard Green, dll's are loaded from both directories. The patches directory just gets cleared out on upgrade of Identity Broker (after v5.0), so it's ideal for this sort of stuff.

Richard Green 9 years ago

Thanks Adam, that's good to know

As discussed, I've attached network traces of the export transaction with the BCE-356 patch applied and removed. There are 2 updates going out in each case.

Following are some details on the connectors and fields involved in the transactions:

Connector (ID = 435e6a2e-3cfa-41d4-aa8e-cba1f3e8d124, Name = SARa)
Adapter (ID = 87653f10-7115-4fd6-bf46-04addfab67b3, Name = SARa User Access)

Key Field: ID ;Type: Integer
Connector Field: End_x0020_Date ;Type: Date
Adapter Field: Endx0020Date

Update 1:
DN: CN=Bob Willis
Key: 89
Endx0020Date: Export Null (Old value: 2015-04-16T00:00:00.000)

Update 2:
DN: CN=Don Bradman
Key: 94
Endx0020Date: Export Null (Old value: 2015-04-16T00:00:00.000)

I'll also attach the connector and adapter config files for good measure.

Adam van Vliet 9 years ago

Thanks Richard Green. Your trace showed that the connector isn't sending out the field being nulled out, which from memory was the right thing to do and was tested. However, I did come across http://sharepoint.stackexchange.com/questions/3009/updatelistitems-wont-let-you-reset-field-value-to-empty which suggested the field should be sent but empty. I've attached Unify.Connectors.Microsoft.SharePoint.zip that does just that.

Please let me know how it goes.

Richard Green 9 years ago

Cheers Adam,

Does this still require the previous patch installed also? Or did the traces show no difference?

Adam van Vliet 9 years ago

I don't think so, the trace didn't appear any different.

Richard Green 9 years ago

Woohoo - confirming that fix has worked Thanks Adam.

Does this mean a connector repackaging?

Bob Bradley 9 years ago

Thanks Adam van Vliet - Richard Green just reported this is now working! Will leave this for Richard Green to mark as resolved once deployed to DEV/TEST.

Adam van Vliet 9 years ago

I'll need to schedule a release. Thanks.

Bob Bradley 9 years ago

Rolled back patch in TEST due to following error which occurred for all 1000 export attempts:

20150515,00:39:34,UNIFY Identity Broker,Connector,Warning,"Add entities to connector failed.
Add entities [Count:1] to connector SARa failed with reason Input string was not in a correct format.. Duration: 00:00:00
Error details:
System.FormatException: Input string was not in a correct format.
   at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at Unify.Communicators.Moss2007ListWebServiceCommunicator.GetBatch(IValueAdapter`2 valuesToJoinedLookupAdapter, IEnumerable`1 data)
   at Unify.Communicators.Moss2007ListWebServiceCommunicator.ModifyListItemAndReturnUpdates(IEnumerable`1 data, IMoss2007ListConnectorInformation connectorInformation)
   at Unify.Communicators.Moss2007ListWebServiceCommunicator.ModifyListItemAndReturnId(IEnumerable`1 data, IMoss2007ListConnectorInformation connectorInformation)
   at Unify.Connectors.MossListConnector.SaveEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.EventNotifierAddingConnectorDecorator.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& backwardAdapterEntities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportAdd(IAdapter adapter, IAdapterEntitySaveChange pendingAdd)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)
   at SyncInvokeExportChanges(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)",Normal
20150515,00:39:34,UNIFY Identity Broker,Adapter,Warning,"Adapter request to add entity to adapter space failed.
Bob Bradley 9 years ago

Restored DLLs allowed exports to work as before - but with the fix for nulling of the date fields still pending again

Adam van Vliet 9 years ago

Released v4.1.1 RC3 at SUBIDBSP:Downloads. Please let me know how it goes.

Richard Green 9 years ago

Hey Adam,

Just got this patch installed in the Origin DEV environment. Testing some export adds produced the following exception. The id field mentioned is the key field - this is not provided on export, and I believe is generated by SharePoint (I will confirm this with Bob).

Add entities to connector failed.Add entities [Count:1] to connector SARa failed with reason The id of <empty> could not be parsed to an integer. Correct the id before performing the export.. Duration: 00:00:00Error details:System.Exception: The id of <empty> could not be parsed to an integer. Correct the id before performing the export. at Unify.Communicators.Moss2007ListWebServiceCommunicator.GetBatch(IValueAdapter`2 valuesToJoinedLookupAdapter, IEnumerable`1 data) at Unify.Communicators.Moss2007ListWebServiceCommunicator.ModifyListItemAndReturnUpdates(IEnumerable`1 data, IMoss2007ListConnectorInformation connectorInformation) at Unify.Communicators.Moss2007ListWebServiceCommunicator.ModifyListItemAndReturnId(IEnumerable`1 data, IMoss2007ListConnectorInformation connectorInformation) at Unify.Connectors.MossListConnector.SaveEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.EventNotifierAddingConnectorDecorator.AddEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& backwardAdapterEntities) at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity) at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportAdd(IAdapter adapter, IAdapterEntitySaveChange pendingAdd) at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter) at SyncInvokeExportChanges(Object , Object[] , Object[] ) at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc) at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
Adam van Vliet 9 years ago

Hi Richard,

Yes it's generated by SharePoint, but will fail with this error if string.Empty is provided. Could you check the logic to make sure that's not happening?

Richard Green 9 years ago

Hey Adam,

Confirming that an empty string is not being exported in this field. I've also rolled back to the original connector and confirmed the record exported as is with that version.

Adam van Vliet 9 years ago

Hi Richard,

I see the issue. It was relying on some logic higher up to return null, but it was converting to string.Empty. Sorry about that. New version available.

Thanks.

Adam van Vliet 9 years ago

Actually, I think I may have broken the fix to be able to clear out values. Please wait whilst I fix that up.

Adam van Vliet 9 years ago

Okay, good to go now. Thanks.

Richard Green 9 years ago

Woohoo - a new error to replace the last one. And this looks like a really useful one

Let me know if you need me to provide a Trace to assist with this one.

Add entities to connector failed.Add entities [Count:1] to connector SARa failed with reason Exception of type 'Microsoft.SharePoint.SoapServer.SoapServerException' was thrown.. Duration: 00:00:00.0469212Error details:System.Web.Services.Protocols.SoapException: Exception of type 'Microsoft.SharePoint.SoapServer.SoapServerException' was thrown. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at Unify.Communicators.ListService.Lists.UpdateListItems(String listName, XmlNode updates) at Unify.Communicators.Moss2007ListWebServiceCommunicator.ModifyListItemAndReturnUpdates(IEnumerable`1 data, IMoss2007ListConnectorInformation connectorInformation) at Unify.Communicators.Moss2007ListWebServiceCommunicator.ModifyListItemAndReturnId(IEnumerable`1 data, IMoss2007ListConnectorInformation connectorInformation) at Unify.Connectors.MossListConnector.SaveEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.EventNotifierAddingConnectorDecorator.AddEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& backwardAdapterEntities) at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity) at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportAdd(IAdapter adapter, IAdapterEntitySaveChange pendingAdd) at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter) at SyncInvokeExportChanges(Object , Object[] , Object[] ) at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc) at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
Adam van Vliet 9 years ago

Yes please Richard Green, a trace would be helpful.

Richard Green 9 years ago

Attached a Network Trace of the Soap Exception.

Adam van Vliet 9 years ago

Thanks Richard Green, unfortunately there's not much more detail in the trace. Have any settings on the list changed? Also, I noticed that the connector is now exporting a bunch of new fields, is there anything special about them (e.g. read-only)? If so, I can try removing them by only exporting non-read-only fields?

Recent traced export:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <soap:Body>
        <UpdateListItems xmlns="http://schemas.microsoft.com/sharepoint/soap/">
            <listName>User Access</listName>
            <updates>
                <Batch OnError="Continue" ListVersion="1" ViewName="" xmlns="">
                    <Method ID="1" Cmd="New">
                        <Field Name="ContentTypeId"/>
                        <Field Name="Title">Network and Email</Field>
                        <Field Name="_ModerationComments"/>
                        <Field Name="File_x0020_Type"/>
                        <Field Name="Start_x0020_Date">2015-04-15</Field>
                        <Field Name="End_x0020_Date"/>
                        <Field Name="Employee_x0020_Type">Permanent</Field>
                        <Field Name="User_x0020_Name">Richard Hadlee</Field>
                        <Field Name="User">-1;#Richard Hadlee</Field>
                        <Field Name="Status">Active</Field>
                        <Field Name="Modified"/>
                        <Field Name="Created"/>
                        <Field Name="Author"/>
                        <Field Name="Editor"/>
                        <Field Name="_HasCopyDestinations"/>
                        <Field Name="_CopySource"/>
                        <Field Name="owshiddenversion"/>
                        <Field Name="WorkflowVersion"/>
                        <Field Name="_UIVersion"/>
                        <Field Name="_UIVersionString"/>
                        <Field Name="Attachments"/>
                        <Field Name="_ModerationStatus"/>
                        <Field Name="InstanceID"/>
                        <Field Name="Order"/>
                        <Field Name="GUID"/>
                        <Field Name="WorkflowInstanceID"/>
                        <Field Name="_Level"/>
                        <Field Name="_IsCurrentVersion"/>
                        <Field Name="ID">New</Field>
                    </Method>
                </Batch>
            </updates>
        </UpdateListItems>
    </soap:Body>
</soap:Envelope>

Previous traced export:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <soap:Body>
        <UpdateListItems xmlns="http://schemas.microsoft.com/sharepoint/soap/">
            <listName>User Access</listName>
            <updates>
                <Batch OnError="Continue" ListVersion="1" ViewName="" xmlns="">
                    <Method ID="1" Cmd="Update">
                        <Field Name="ContentTypeId">0x0100E2E95B89E32D034EA2EB9455CCECC8CF</Field>
                        <Field Name="Editor">svc-fim-idb</Field>
                        <Field Name="User_x0020_Name">Don Bradman</Field>
                        <Field Name="Order">9400</Field>
                        <Field Name="WorkflowVersion">1</Field>
                        <Field Name="_UIVersion">512</Field>
                        <Field Name="User">-1;#Don Bradman</Field>
                        <Field Name="Created">2015-05-01T05:24:09Z</Field>
                        <Field Name="_Level">1</Field>
                        <Field Name="_ModerationStatus">0</Field>
                        <Field Name="_IsCurrentVersion">True</Field>
                        <Field Name="_UIVersionString">1.0</Field>
                        <Field Name="owshiddenversion">40</Field>
                        <Field Name="Title">Network andEmail</Field>
                        <Field Name="GUID">42df5d1d-bcb5-4bea-80f1-e480735ec2c0</Field>
                        <Field Name="Employee_x0020_Type">Permanent</Field>
                        <Field Name="Author">Bob Bradley</Field>
                        <Field Name="Modified">2015-05-14T00:43:07Z</Field>
                        <Field Name="Attachments">False</Field>
                        <Field Name="Status">Terminated</Field>
                        <Field Name="Start_x0020_Date">2015-04-15</Field>
                        <Field Name="ID">94</Field>
                    </Method>
                </Batch>
            </updates>
        </UpdateListItems>
    </soap:Body>
</soap:Envelope>
Bob Bradley 9 years ago

Adam van Vliet - knowing what I do about SharePoint lists, yes that is very much going to be the problem here.
The values in the first trace appear to be the ones that correspond to FIM export flow rules - the other are indeed going to be internal to SharePoint and should not be set.
Of the following, the ones I have marked with an X (XName) are the ones that I believe we should (continue to be) treating as read only.

                        <Field XName="ContentTypeId">0x0100E2E95B89E32D034EA2EB9455CCECC8CF</Field>
                        <Field XName="Editor">svc-fim-idb</Field>
                        <Field Name="User_x0020_Name">Don Bradman</Field>
                        <Field XName="Order">9400</Field>
                        <Field XName="WorkflowVersion">1</Field>
                        <Field XName="_UIVersion">512</Field>
                        <Field Name="User">-1;#Don Bradman</Field>
                        <Field XName="Created">2015-05-01T05:24:09Z</Field>
                        <Field XName="_Level">1</Field>
                        <Field XName="_ModerationStatus">0</Field>
                        <Field XName="_IsCurrentVersion">True</Field>
                        <Field XName="_UIVersionString">1.0</Field>
                        <Field XName="owshiddenversion">40</Field>
                        <Field Name="Title">Network andEmail</Field>
                        <Field XName="GUID">42df5d1d-bcb5-4bea-80f1-e480735ec2c0</Field>
                        <Field Name="Employee_x0020_Type">Permanent</Field>
                        <Field XName="Author">Bob Bradley</Field>
                        <Field XName="Modified">2015-05-14T00:43:07Z</Field>
                        <Field XName="Attachments">False</Field>
                        <Field Name="Status">Terminated</Field>
                        <Field Name="Start_x0020_Date">2015-04-15</Field>
                        <Field XName="ID">94</Field>

I expect that some special cases above will need to be set on insert (ID=0, GUID?), but not on modify.

I have a change hanging over my head here to change the DN/anchor for this connector to something unique - was thinking of ID but now thinking I need to use GUID (because ID is going to be 0 I think for all new provisioned items). I am using the FIM guid for the GUID property on create.

Adam van Vliet 9 years ago

Thanks Bob Bradley. I'll update the connector to only include items that are not marked as read-only in the connector schema, as well as the ID field and any special cases we come up with.

Bob Bradley 9 years ago

Adam van Vliet - it really comes down to only writing those attributes that FIM is trying to change isn't it (rather than the complete set)? I think we had a similar challenge with the PowerShell connector.

Adam van Vliet 9 years ago

That's true, but it's not something we have direct access to at the connector layer. We have the current state, optionally the previous state (from the connector store), and optionally the previous state (directly from SharePoint). From that we have to figure out what the desired behaviour is. The read-only setting is probably the easiest way to do it, as it becomes a configuration exercise instead of having to do anything more complicated/slower/error prone (retrieving from either IdB or SharePoint).

Adam van Vliet 9 years ago

I've done another release. I did it as a DEV release this time though as I want to get it right before I do up another candidate.

Bob Bradley 9 years ago

Latest version (27/5/2015) is looking good so far, Adam van Vliet and Richard Green . Exported all items just now in DEV, including null values for dates, and am not getting the dreaded exported-not-imported error. Will confirm when we are good for a final release version.

Bob Bradley 9 years ago

Adam van Vliet - please create a release candidate for Origin and alert Richard Green and André van der Westhuizen. The latest one works well thanks, and we could deploy to Origin SIT/UAT/PROD with that, but I suspect you want us to use a new one?

Adam van Vliet 9 years ago

Bob Bradley - released v4.1.1 RTM.

Bob Bradley 9 years ago

Thanks Adam van Vliet. Richard Green - have just copied this down to DEV (C:\FIM Installers) ready to install when you are.

Customer support service by UserEcho