0
Under review

Exporting to SharePoint Orgs results in cd-error after incorrect Parent Org calculation

Ryan Crossingham 2 months ago in UNIFYBroker/Microsoft SharePoint • updated by Beau Harrison (Software Developer) 2 weeks ago 10

Hi There,

Current client is implementing a new HR system which is also authoritative for Org objects.
The Org objects are flowing through to SharePoint and were previously being sourced by Aurion.


During the deployment, the Parent Org calculation was incorrect and exported to SharePoint pointing to the incorrect parent Org. This resulted in a some MIM errors as you would expect (cd-error) - No further Event Viewer logs/info

As a result, the idMParentProfileReference attribute in IdentityBroker used to provide the Parent Org is now NULL for some objects. The good thing is that the Org structure does not look it has been updated or changed at this point.

We now have the correct Parent Org structure in MIM ready to export but the SharePoint Org exports continue to fail with a cd-error.

Referencing the idB for SharePoint prerequisites KB, it sounds as if these need to be filled manually the first time for these exports to succeed. 

Example screenshot of before and after attached.

Before

After



I understand that the idMParentProfileReference needs to be filled in order for the Org structure to be managed as shown the example from the Prerequisites KB: Example below:



In order to manage SharePoint 2010 organization profiles, a field must be manually added to the SharePoint schema, and populated for any users who exist prior to enabling Identity Broker. This is required because SharePoint uses its own internal Record Id for resolving the parent reference with SharePoint, and this field cannot be set externally unless the corresponding SharePoint identifier for the parent profile is used. This is typically an organization unit code or identifier. This field should be either a string, integer, or distinguished name type in SharePoint, and will need to be appropriately configured in the Microsoft SharePoint 2010 Organization Profile Connector schema. The default connector configuration assumes a name of IdmProfileReference for this field.
In order to successfully provision and update hierarchy information for organization profiles, the connector requires this field containing the value of profile's reference in the identity management solution, and an additional field containing the profile's parent reference in a DN format (which does not need to be added to SharePoint). Refer to Microsoft SharePoint 2010 Organization Profile Connector for more information.

Does the idM Profile Reference need to be filled manually the first time / if it is NULL? 

OR

Is it expected that MIM can write to it freely?

I'm just trying to get an understanding of why SharePoint wont accept the structure i'm exporting.
I have ensured that the DN format is correct and written as it was previously - I believe this has to do with the fact that incorrect parent Org DNs were exported in the first instance.

Imports of a test connector (Copy of the original connector) Show the following in Logs:

Request to import all entities from connector SP TEST.",Normal
20190620,00:39:51,UNIFY Identity Broker,Connector,Information,"Import all entities from connector completed.
Import all entities from connector SP TEST return 868 entities. Duration: 00:00:00",Normal
20190620,00:39:51,UNIFY Identity Broker,Connector Processor,Information,"Connector Processing started.
Connector Processing started for connector SP TEST (page 1)",Normal
20190620,00:39:51,UNIFY Identity Broker,Connector engine,Information,"Request to get the enabled state of the selected connector.
Request to get the enabled state of the 8955f94f-4373-424e-a502-e8d8bc2c1fd4 connector started.",Verbose
20190620,00:39:51,UNIFY Identity Broker,Connector engine,Information,"Request to get the enabled state of the selected connector.
Request to get the enabled state of the 8955f94f-4373-424e-a502-e8d8bc2c1fd4 connector completed. Duration: 00:00:00",Verbose
20190620,00:39:51,UNIFY Identity Broker,Connector Processor,Information,"Connector processing failed.
Connector Processing page 1 for connector SP TEST failed with reason The key has been duplicated.. Duration: 00:00:00.1875685.
Error details:
System.ArgumentException: The key has been duplicated.
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue`1 arg1)
at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_0.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()",Normal
20190620,00:39:51,UNIFY Identity Broker,Change detection engine,Error,"Change detection engine import all items failed.
Change detection engine import all items for connector SP TEST failed with reason An error occurred while evaluating a task on a worker thread. See the inner exception details for information.. Duration: 00:00:11.2399393
Error details:
Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentException: The key has been duplicated.
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue`1 arg1)
at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_0.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()
--- End of inner exception stack trace ---
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.WaitForCompletedThreads()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPool[T](IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.b__0()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal
20190620,00:39:51,UNIFY Identity Broker,Void CheckForException(),Error,"Unify.Framework.DesignPatterns:
Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentException: The key has been duplicated.
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue`1 arg1)
at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_0.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()
--- End of inner exception stack trace ---
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.WaitForCompletedThreads()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPool[T](IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.b__0()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Diagnostic


I'm not sure if this is directly related to this issue but this looks as though a NULL key has been added and may need to fixed up at the source.

Config copy attached

Any assistance would be appreciated.


Ryan

Affected Versions:
Fixed by Version:
+1
Under review

Hi Ryan

The IdMProfileReference field must be set for all organizations on connector import. Orgs that already exist before Broker is setup must have this field manually so it's there for the first import. New orgs should be provisioned with this field set.

The IdMParentProfileReference field is calculated on import by finding the IdMProfileReference of the org with the RecordId matching the first orgs ParentRecordId. When the connector is creating or updating an org, if the IdMParentProfileReference is set, the parent IdMProfileReference is extracted from the DN and used to look up a RecordId to set as the updated entities ParentRecordId.

Have you tried clearing and reimporting the org connector before running the corrected MIM export? The old values in the connector may be influencing the ParentRecordId lookup.

Can you provide your config?

Thanks Beau.

Sorry I thought I had attached it.
See below

Config.zip

Hey Ryan, what appears in the Broker log for the failing exports?

Unfortunately, I can't see anything error related at all.
I will run another export attempt when the service becomes available and send the logs. I will backup the current log to dial down the noise.

Currently whats happening is when the export run its stopping the SharePoint WCF Service and I need to run a restart of SharePoint to get it back up again -  Which means I can only test this out of hours.

Attempting again today I did the following:

    • Cleared the idB Connector and reimported - Ensured the Adapter cleared and reflected the new connector data
    • Ensured the Org data being exported is correct
    • Executed the Export
    • Same cd-error in MIM with nothing noteworthy in logs

    I'm wondering if this is related to the current data in SharePoint and less to do with the data being exported. Is there any additional logging I can enable to tease out an error?

      Hi Beau,

      After limiting down the exports to one at a time an error has presented itself.

      See Event viewer logs attached.

      SharePoint MA Errors.evtx

      Any idea what this could be related to?

      Hi Ryan, have you made any progress on this one?

      Hi Beau,

      Can you confirm something for me?

      Should I be exporting the ParentRecordID or the IdMParentProfileReference from MIM?

      You mentioned that  the IdMParentProfileReference is calculated by the connector on import. I'm currently writing the IdMParentProfileReference from my MA and not writing to the ParentRecordID.

      Is this where I am going wrong?

      For context, this is the process:

      IdMParentProfileReferenceField is a DN, format is OU=idMParentId,OU=AdapterContainerName,DC=IdentityBroker

      idMParentId is looked up using the ParentOrganizationProfileId from SharePoint (ParentOrganizationProfileId is also set)

      If IdMParentProfileReferenceField is being set correctly on import into UNIFYBroker then it means it is doing the lookup correctly (using the stored value context to store this state, you can see it in the db for reference).

      On update, the first OU value from IdMParentProfileReferenceField is used to match on the IdMReferenceField of the context entities to set the value of the ParentOrganizationId (which is taken from the RecordId of the other entity). So you are correct on setting the IdMParentProfileReferenceField, it should be using that to figure out the target RecordId.

      With regards to the extra logging, I'll speak with Matt (Beau is on leave), in the meantime you could rawcap the LDAP response and/or WCF trace the response from SharePoint.

      I can now run my exports as required and reproduce the error easily.

      See errors below - I have applied the provided "Unify.IdentityBroker.FIMAdapter.dll".
      Can't see any additional log info though.

      The management agent controller encountered an unexpected error.

      "BAIL: MMS(696): extensionmanager.cpp(620): 0x80230703 (The extension threw an exception.)
      BAIL: MMS(696): extensionmanager.cpp(2648): 0x80230703 (The extension threw an exception.)
      BAIL: MMS(696): export.cpp(2150): 0x80230703 (The extension threw an exception.)
      BAIL: MMS(696): export.cpp(521): 0x80230703 (The extension threw an exception.)
      BAIL: MMS(696): ..\cntrler.cpp(9848): 0x80230703 (The extension threw an exception.)
      BAIL: MMS(696): ..\cntrler.cpp(8569): 0x80230703 (The extension threw an exception.)
      Forefront Identity Manager 4.4.1459.0"

      The extensible extension returned an unsupported error.
      The stack trace is:

      "System.InvalidCastException: Unable to cast object of type 'Unify.Product.IdentityBroker.Asn1Sequence' to type 'Unify.Product.IdentityBroker.ILdapResult'.
      at Unify.Product.IdentityBroker.BulkUpdateRequest.UpdateResponse.<>c.b__2_1(Asn1Sequence error)
      at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
      at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
      at Unify.Product.IdentityBroker.BulkUpdateRequest.UpdateResponse.get_Errors()
      at Unify.Product.IdentityBroker.BulkUpdateRequest.ResponseToResults(LdapMessage response, UpdateRequest page)
      at System.Linq.Enumerable.d__61`3.MoveNext()
      at System.Linq.Enumerable.d__17`2.MoveNext()
      at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
      at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
      at Unify.Product.IdentityBroker.BulkUpdateRequest.Send(Func`2 send, Func`2 recv)
      at Unify.Product.IdentityBroker.LdapConnection.SendRequest(ILdapRequest request)
      at Unify.Product.IdentityBroker.LdapConnectionProxy.SendRequest(ILdapRequest request)
      at Unify.Product.IdentityBroker.ExportProxy.GetBulkRequestResult(BulkUpdateRequest request)
      at Unify.Product.IdentityBroker.ExportProxy.BulkExportEntries(IList`1 csentries)
      at Unify.Product.IdentityBroker.ExportProxy.Export(IList`1 csentries)
      at Unify.Product.IdentityBroker.UnifyLdapConnectorTypeProxy.PutExportEntries(IList`1 csentries)
      at Unify.Product.IdentityBroker.UnifyLdapConnector.PutExportEntries(IList`1 csentries)
      Forefront Identity Manager 4.4.1459.0"


      The Export attempt data below:

      EXPORT ATTEMPT.XML

      Hi Ryan

      Here is a patched MA that should stop the above error being thrown and output a more helpful error message. Please backup and replace the MA dll located at %ProgramFiles%\Microsoft Identity Integration Server/Extensions.

      Unify.IdentityBroker.FIMAdapter.dll

      It would also be great if you could run a LDAP capture using rawcap to help diagnose this issue.