Record with a duplicate key in the CSV file backing a CSV connector created when a link has outgoing join criteria fields not in the connector's key and only non-key field values are changing

When a locker entity is updated with changes to a link's outgoing join criteria field values but no change to any fields part of the underlying CSV connector's key then a duplicate record is written to the CSV file (which causes subsequent imports to fail).

This bug was previously reported in https://voice.unifysolutions.net/en/communities/6/topics/4209-locker-entity-not-deprovisioned-when-adapter-entity-join-field-values-change# but that ticket covers a number of different problems experienced and so this simple ticket is being created for improved clarity.

Notes
This bug has only been confirmed for the CSV connector at this time.

No error is logged during the outgoing changes sync:

20210208,02:46:18,UNIFYBroker,SyncEngine,Information,"Request to sync locker to adapter started.
Synchronization job started syncing 1 changes on the 'out' link from the locker to adapter. Job ID: 2c5fdfa5-ae77-4bb8-b109-10ca7b2810ff",Normal
20210208,02:46:18,UNIFYBroker,Adapter,Information,"Adapter request to add entities for adapter space.
Adapter request to add entities [Count:1] for adapter out (054763ed-db86-4092-bbd9-02b3b8e9736e).",Normal
20210208,02:46:18,UNIFYBroker,Connector,Information,"Request to add entities to connector.
Request to add entities [Count:1] to connector out.",Normal
20210208,02:46:18,UNIFYBroker,Connector,Information,"Add entities to connector completed.
Add entities [Count:1] to connector out reported 1 entities saved, 0 failed. Duration: 00:00:00.1622925",Normal
20210208,02:46:18,UNIFYBroker,Adapter,Information,"Adapter added entities to adapter space.
Adapter added [Count:1] entities (1 successful) to adapter out (054763ed-db86-4092-bbd9-02b3b8e9736e). Duration: 00:00:00.1622925",Normal
20210208,02:46:18,UNIFYBroker,Connector,Information,"Request to update entity to connector.
Request to update entities [Count:0] to connector out.",Normal
20210208,02:46:19,UNIFYBroker,Connector,Information,"Update entities to connector completed.
Update entities 0 to connector out reported 0 entities saved, 0 failed. Duration: 00:00:00.1199350",Normal
20210208,02:46:19,UNIFYBroker,SyncEngine,Information,"Request to sync locker to adapter completed.
Synchronization job completed syncing 1 changes on the 'out' link from the locker to adapter. Delayed: 0 Incomplete: 0 Denied: 0 Job ID: 2c5fdfa5-ae77-4bb8-b109-10ca7b2810ff Duration: 00:00:00.4020608",Normal
20210208,02:46:23,UNIFYBroker,Change detection engine,Information,"Change detection engine unscheduled started.
Change detection engine unscheduled for connector out started.",Normal
20210208,02:46:23,UNIFYBroker,Change detection engine,Information,"Change detection engine unscheduled completed.
Change detection engine unscheduled for connector out completed. Duration: 00:00:00.0219985",Normal
20210208,02:46:27,UNIFYBroker,Change detection engine,Information,"Started processing changes register items.
Started processing changes register items for connector out.",Normal
20210208,02:46:27,UNIFYBroker,Change detection engine,Information,"Changes register item processing completed.
Changes register item processing on connector out completed. Duration: 00:00:00.0950008",Normal
20210208,02:46:31,UNIFYBroker,Adapter,Information,"Request to reflect change entities of the adapter.
Request to reflect change entities of the out (054763ed-db86-4092-bbd9-02b3b8e9736e) adapter completed with 0 adds, 0 updates and 0 deletes across 1 pages. Duration: 00:00:00.0500007",Normal

No error is logged for the confirming incoming changes sync prior to the connector import operation:

20210208,02:51:19,UNIFYBroker,SyncEngine,Information,"Request to sync adapter to locker started.
Synchronization job started syncing 1 changes on the 'out' link from the adapter to locker. Job ID: 828030c1-7dda-4b15-9572-f8a99b31e463",Normal
20210208,02:51:19,UNIFYBroker,SyncEngine,Information,"Request to sync adapter to locker completed.
Synchronization job completed syncing 1 changes on the 'out' link from the adapter to locker. Delayed: 0 Incomplete: 0 Denied: 0 Job ID: 828030c1-7dda-4b15-9572-f8a99b31e463 Duration: 00:00:00.1417907",Normal

The subsequent connector import error is:

20210208,02:53:19,UNIFYBroker,Change detection engine,Information,"Change detection engine import all items started.
Change detection engine import all items for connector out started.",Normal
20210208,02:53:19,UNIFYBroker,Connector,Information,"Request to import all entities from connector.
Request to import all entities from connector out.",Normal
20210208,02:53:19,UNIFYBroker,Connector,Information,"Import all entities from connector completed.
Import all entities from connector out return 2 entities. Duration: 00:00:00",Normal
20210208,02:53:19,UNIFYBroker,Connector Processor,Information,"Connector Processing started.
Connector Processing started for connector out (page 1)",Normal
20210208,02:53:19,UNIFYBroker,Connector Processor,Information,"Connector processing failed.
Connector Processing page 1 for connector out failed with reason The key 1 has been duplicated.. Duration: 00:00:00.3093309.
Error details:
System.ArgumentException: The key 1 has been duplicated.
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue`1 arg1)
at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_1.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()",Normal
20210208,02:53:19,UNIFYBroker,Change detection engine,Error,"Change detection engine import all items failed.
Change detection engine import all items for connector out failed with reason An error occurred while evaluating a task on a worker thread. See the inner exception details for information.. Duration: 00:00:00.3583489
Error details:
Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentException: The key 1 has been duplicated.
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue`1 arg1)
at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_1.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()
--- End of inner exception stack trace ---
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.WaitForCompletedThreads()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPool[T](IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.b__0()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal