0
Answered

An item with the same key has already been added

Carol Wapshere 6 years ago in PowerShell connector updated by Beau Harrison (Senior Product Software Engineer) 6 years ago 3

I have just upgraded IdB in TEST to 5.2, and migrated in the Connector and Adapter files from Dev. Dev was already on 5.2 and all connectors are working.

In TEST all of the new Connectors (names "PowerShell HomeFolder*" and "PowerShell MemberOf*") are failing with the error below. The "Powershell Exchange*" connectors work fine (though they already pre-existing in IdB 5.1 before I upgraded to 5.2).

The Connector config file is the same as the one I sent with the previous question. While the error looks very similar to that one it can't be the same - that was a duplicate schema mapping in the connector config, but Powershell connectors don't have schema mapping.

My Import scripts drop a full log of all entity values before running the $entity.Create ... $entity.Commit loop. There are no duplicate sAMAccountNames.

Note that when I did the IdB database upgrade in Test I removed the final lines from the script as told to do here. I don't seem to have had any problems with this in Dev, but thought it worth mentioning.


Change detection engine import all items failed.
Change detection engine import all items for connector PowerShell HomeFolder NMI failed with reason An error occurred while evaluating a task on a worker thread. See the inner exception details for information.. Duration: 00:01:37.5326976
Error details:
Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
at Unify.Product.IdentityBroker.Repository.EntityLinqQueryConverterUtilitiesBase`4.GetCollectionKeyData(TEntityKey key, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.EntitySingleValueDataUtilityBase`2.CreateEntityValue(TEntityKey key, IValue value, IEntityCollectionKeyUtility`1 collectionKeyUtility, EntityDataSet set, __EntityInsertRow row, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.ConvertEntityValueToDataValue(KeyValuePair`2 entityValueAndKey, __EntityInsertRow row, EntityDataSet entityDataSet, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.<>c__DisplayClass31_0.<convertitemtovalues>b__0(KeyValuePair`2 entityValueAndKey)
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.<selectmanyiterator>d__16`2.MoveNext()
at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.InsertItems(ISet`1 addedItems, EntityDataContext sourceContext, SqlConnection connection)
at Unify.Framework.Data.LinqContextConversionBase`4.SubmitChanges()
at Unify.Product.IdentityBroker.SaveChangedEntitiesTransformationUnit.Transform(IDictionaryTwoPassDifferenceReport`4 input)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_0.<performchangedetection>b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()
--- End of inner exception stack trace ---
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.<run>b__0()
at Unify.Framework.<span class="redactor-selection-marker" id="selection-marker-1"></span>AsynchronousJobExecutor.PerformJobCallback(Object state)
</run></performchangedetection></selectmanyiterator></convertitemtovalues>

Answer

Answer
Answered

For future reference, this issue is caused by entries in the CollectionKey table with the same Caption field value. The duplicate captions, produced by a defect in Identity Broker v5.1, cause exceptions to be thrown in several areas of the application after performing an upgrade to Identity Broker v5.2 which assume these values to be unique.

The simplest solution for this issue would be running the database clear script found in the <InstallDir>/Database directory. If this is not possible or desirable, attempt to run the script I provided below which clears the CollectionKey table of all unused entries and may resolve this issue. If the issue persists at this point a script or tool can be provided suitable to the specific environment to more directly correct the database state.

I think it has something to do with duplicate schema names in the database. How do I fix that? If I run this:

select ck.[CollectionKeyId], ck.Caption from [dbo].[CollectionKey] ck
join
(
  select [Caption] from [dbo].[CollectionKey]
  group by Caption
  having count(Caption) > 1
) dup
on ck.Caption = dup.Caption
order by ck.Caption asc 
I get this:
CollectionKeyId Caption                    
--------------- -------                    
            203 HomePostcode               
            238 HomePostCode               
            114 OrganisationUnitDescription
            100 OrganisationUnitDescription
             99 OrganisationUnitLevel      
            110 OrganisationUnitLevel      
             97 OrganisationUnitNumber     
            109 OrganisationUnitNumber     
             98 OrganisationUnitStatus     
            115 OrganisationUnitStatus     
             95 OrgUnit1ID                 
            101 OrgUnit1ID                 
            102 OrgUnit2ID                 
            111 OrgUnit2ID                 
            103 OrgUnit3ID                 
            112 OrgUnit3ID                 
            104 OrgUnit5ID                 
            113 OrgUnit5ID                 
            105 OrgUnit6ID                 
             96 OrgUnit6ID                 
            106 OrgUnit7ID                 
            116 OrgUnit7ID                 
            194 OrgUnit7Name               
            192 OrgUnit7Name               
            195 OrgUnit7Ref                
            191 OrgUnit7Ref                
            107 OrgUnit8ID                 
            117 OrgUnit8ID                 
            108 OrgUnit9ID                 
            118 OrgUnit9ID                 
            128 OSUserId                   
             50 OsUserId                   
            216 Postcode                   
            243 PostCode                   
            126 UserId                     
            224 UserID 
I noticed that the value in OsUserID has disappeared in the adapter in Test and there's now also a reflection error for that adapater. The connector still has the value, that particular attribute is not used in any other connector, and is not renamed in the adapter.
Under review

Hi Carol, 

You're correct. Duplicate captions are the cause which probably originated in the previous version. If you haven't already, try clearing the CollectionKey table. This will cause IdB to re-populate it, hopefully with no duplicates. Let me know if you notices v5.2 generating duplicates and continuing to raise this exception.

Actually, clearing won't work due to existing entities relying on that collection key id. You will need to perform a selective delete where the CollectionKeyId is not references in the EntityValue or EntityValueOriginTables. For example:

DELETE FROM CollectionKey 
WHERE CollectionKeyId NOT IN (SELECT CollectionKeyId FROM EntityValue)
AND (CollectionKeyId NOT IN (SELECT CollectionKeyId FROM EntityValueOrigin)
OR CollectionKeyId NOT IN (SELECT SourceCollectionKeyId FROM EntityValueOrigin))

Once done, run your query from above again and there will hopefully the only duplicates will be have different casing.

Answer
Answered

For future reference, this issue is caused by entries in the CollectionKey table with the same Caption field value. The duplicate captions, produced by a defect in Identity Broker v5.1, cause exceptions to be thrown in several areas of the application after performing an upgrade to Identity Broker v5.2 which assume these values to be unique.

The simplest solution for this issue would be running the database clear script found in the <InstallDir>/Database directory. If this is not possible or desirable, attempt to run the script I provided below which clears the CollectionKey table of all unused entries and may resolve this issue. If the issue persists at this point a script or tool can be provided suitable to the specific environment to more directly correct the database state.