When a locker entity is updated with changes to a link's outgoing join criteria field values but no change to any fields part of the underlying CSV connector's key then a duplicate record is written to the CSV file (which causes subsequent imports to fail).

This bug was previously reported in https://voice.unifysolutions.net/en/communities/6/topics/4209-locker-entity-not-deprovisioned-when-adapter-entity-join-field-values-change# but that ticket covers a number of different problems experienced and so this simple ticket is being created for improved clarity.

Notes
This bug has only been confirmed for the CSV connector at this time.

No error is logged during the outgoing changes sync:

20210208,02:46:18,UNIFYBroker,SyncEngine,Information,"Request to sync locker to adapter started.
Synchronization job started syncing 1 changes on the 'out' link from the locker to adapter. Job ID: 2c5fdfa5-ae77-4bb8-b109-10ca7b2810ff",Normal
20210208,02:46:18,UNIFYBroker,Adapter,Information,"Adapter request to add entities for adapter space.
Adapter request to add entities [Count:1] for adapter out (054763ed-db86-4092-bbd9-02b3b8e9736e).",Normal
20210208,02:46:18,UNIFYBroker,Connector,Information,"Request to add entities to connector.
Request to add entities [Count:1] to connector out.",Normal
20210208,02:46:18,UNIFYBroker,Connector,Information,"Add entities to connector completed.
Add entities [Count:1] to connector out reported 1 entities saved, 0 failed. Duration: 00:00:00.1622925",Normal
20210208,02:46:18,UNIFYBroker,Adapter,Information,"Adapter added entities to adapter space.
Adapter added [Count:1] entities (1 successful) to adapter out (054763ed-db86-4092-bbd9-02b3b8e9736e). Duration: 00:00:00.1622925",Normal
20210208,02:46:18,UNIFYBroker,Connector,Information,"Request to update entity to connector.
Request to update entities [Count:0] to connector out.",Normal
20210208,02:46:19,UNIFYBroker,Connector,Information,"Update entities to connector completed.
Update entities 0 to connector out reported 0 entities saved, 0 failed. Duration: 00:00:00.1199350",Normal
20210208,02:46:19,UNIFYBroker,SyncEngine,Information,"Request to sync locker to adapter completed.
Synchronization job completed syncing 1 changes on the 'out' link from the locker to adapter. Delayed: 0 Incomplete: 0 Denied: 0 Job ID: 2c5fdfa5-ae77-4bb8-b109-10ca7b2810ff Duration: 00:00:00.4020608",Normal
20210208,02:46:23,UNIFYBroker,Change detection engine,Information,"Change detection engine unscheduled started.
Change detection engine unscheduled for connector out started.",Normal
20210208,02:46:23,UNIFYBroker,Change detection engine,Information,"Change detection engine unscheduled completed.
Change detection engine unscheduled for connector out completed. Duration: 00:00:00.0219985",Normal
20210208,02:46:27,UNIFYBroker,Change detection engine,Information,"Started processing changes register items.
Started processing changes register items for connector out.",Normal
20210208,02:46:27,UNIFYBroker,Change detection engine,Information,"Changes register item processing completed.
Changes register item processing on connector out completed. Duration: 00:00:00.0950008",Normal
20210208,02:46:31,UNIFYBroker,Adapter,Information,"Request to reflect change entities of the adapter.
Request to reflect change entities of the out (054763ed-db86-4092-bbd9-02b3b8e9736e) adapter completed with 0 adds, 0 updates and 0 deletes across 1 pages. Duration: 00:00:00.0500007",Normal

No error is logged for the confirming incoming changes sync prior to the connector import operation:

20210208,02:51:19,UNIFYBroker,SyncEngine,Information,"Request to sync adapter to locker started.
Synchronization job started syncing 1 changes on the 'out' link from the adapter to locker. Job ID: 828030c1-7dda-4b15-9572-f8a99b31e463",Normal
20210208,02:51:19,UNIFYBroker,SyncEngine,Information,"Request to sync adapter to locker completed.
Synchronization job completed syncing 1 changes on the 'out' link from the adapter to locker. Delayed: 0 Incomplete: 0 Denied: 0 Job ID: 828030c1-7dda-4b15-9572-f8a99b31e463 Duration: 00:00:00.1417907",Normal

The subsequent connector import error is:

20210208,02:53:19,UNIFYBroker,Change detection engine,Information,"Change detection engine import all items started.
Change detection engine import all items for connector out started.",Normal
20210208,02:53:19,UNIFYBroker,Connector,Information,"Request to import all entities from connector.
Request to import all entities from connector out.",Normal
20210208,02:53:19,UNIFYBroker,Connector,Information,"Import all entities from connector completed.
Import all entities from connector out return 2 entities. Duration: 00:00:00",Normal
20210208,02:53:19,UNIFYBroker,Connector Processor,Information,"Connector Processing started.
Connector Processing started for connector out (page 1)",Normal
20210208,02:53:19,UNIFYBroker,Connector Processor,Information,"Connector processing failed.
Connector Processing page 1 for connector out failed with reason The key 1 has been duplicated.. Duration: 00:00:00.3093309.
Error details:
System.ArgumentException: The key 1 has been duplicated.
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue`1 arg1)
at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_1.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()",Normal
20210208,02:53:19,UNIFYBroker,Change detection engine,Error,"Change detection engine import all items failed.
Change detection engine import all items for connector out failed with reason An error occurred while evaluating a task on a worker thread. See the inner exception details for information.. Duration: 00:00:00.3583489
Error details:
Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentException: The key 1 has been duplicated.
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue`1 arg1)
at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_1.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()
--- End of inner exception stack trace ---
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.WaitForCompletedThreads()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPool[T](IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.b__0()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal

For an adapter entity already joined to a locker entity, if any of the join field values change but the entity ID remains unchanged (i.e. none of the underlying connector's keys change) then the previously joined locker entity is retained and a new one is provisioned instead (or presumably joined, if there was a matching entity already present in the locker).

This bug was previously reported in https://voice.unifysolutions.net/en/communities/6/topics/4209-locker-entity-not-deprovisioned-when-adapter-entity-join-field-values-change# but that ticket covers a number of different bugs experienced and so this simple ticket is being created for improved clarity.

When a link has incoming deprovisioning configured and there is a change to fields of an existing adapter entity which are part of the join criteria then the existing locker entity is not deprovisioned, and a new entity for the new join field values is either provisioned or joined (if there is already a locker with matching join field values).

This differs from the behaviour when an adapter entity is deleted, which correctly triggers locker deprovisioning.

This is something I've mentioned previously in https://voice.unifysolutions.net/en/communities/6/topics/4204-changing-the-value-of-a-links-join-criteria-field-causes-existing-locker-entity-to-become-unjoined because Matt asked me to note it in passing, but now it's actually causing a problem with a UNIFYConnect customer implementation so it needs to be escalated as a bug to fix more urgently.

Add support for non-CSV connector types to the Test Harness plugin.

This would help address problems relating to multivalued fields with commas in values.

I have a CSV connector with a date field, and it is expecting US date format (mm/dd/yyyy).  Can this be set to expect Australia format (dd/mm/yyyy) instead, or should I parse it by hand using a PowerShell transform in my adapter?

I updated a multivalue string field in a Adapter (adding two different values to two already present) and the change did not map to the Locker using a Link configured only for Change Polling.

When I ran a Baseline Sync on the Link, the change went through.

I will try to reproduce this.

During his SIT recently Adam updated a multivalue string field in a Locker (removing two values, adding two different values) and the change did not map to the Adapter using a Link configured only for Change Polling.

When I ran a Baseline Sync on the Link, the change went through.

I will try to reproduce this.

When the value of a field that is used in a Link's join criteria changes, that Join is lost and a new one is created (either via provisioning a new locker entity, or - presumably as I have never attempted it - by joining to a different locker entity if one exists that matches the new field value).

While this might appear to be reasonable behaviour at first, the outcome of this is that when the Join criteria field value of an adapter entity changes it ends up leaving an old entity with the previous value behind, and creating a brand new one with all the same values other than the join criteria field, rather than simply updating the previously-joined locker entity based on the mappings.

Consider:

If I changed the Access Package Name in my adapter from "Admins" to "Sysadmins" I might reasonably expect that change to be mapped through to update my existing locker entity.  Instead, I end up with the old, now-unjoined "Admins" locker entity, as well as a newly provisioned "Sysadmins" entity.  This seems somewhat counter-intuitive: it feels like Joins should have more permanence than that.

Setting Incoming Deprovision to True on the link might appear to be a solution to this, but I may well want to retain locker entities normally (i.e. if the adapter entity was genuinely deleted).  And the deletion/recreation of the locker entity is unnecessary anyway.

Adapter "DWH Employee" has a Join to connector "SPOL Suspended Employee".  When an existing entity in that connector updated (to change the value of a boolean attribute) the Join did not re-evaluate on the adapter. Running an Import All on SPOL Suspended Employee did not update the DWH Employee adapter entity, and neither did running an Import All on the DWH Employee adapter's base connector. The join field was only updated when I ran Generate Changes on the DWH Employee adapter directly.

Note: this has the same appearance as ticket #4200 but is happening for a Join transform rather than a Foreign Multivalue Group transform, so it is possible that the same problem may be present across all join-type transforms.

This morning I moved from users from one AD org unit to another, which resulted in their "dn" attribute changing.  One of those users is a manager for other subordinate users, and that user's dn value is retrieved by an adapter Join transform on the subordinate users, and mapped into their "manager" field in AD via their locker records.

After the change was made, the manager's connector and adapter record updated with the new dn field value, but the updated value did not flow through to the subordinate users' "manager" field in the locker.  The link which should be responsible for this is only configured for Change Polling.  When I ran a Baseline Sync, however, the subordinate users' manager fields did update in the locker.

I haven't tried to replicate this problem, but I do not believe it to be typical behaviour.