An item with the same key has already been added
I have just upgraded IdB in TEST to 5.2, and migrated in the Connector and Adapter files from Dev. Dev was already on 5.2 and all connectors are working.
In TEST all of the new Connectors (names "PowerShell HomeFolder*" and "PowerShell MemberOf*") are failing with the error below. The "Powershell Exchange*" connectors work fine (though they already pre-existing in IdB 5.1 before I upgraded to 5.2).
The Connector config file is the same as the one I sent with the previous question. While the error looks very similar to that one it can't be the same - that was a duplicate schema mapping in the connector config, but Powershell connectors don't have schema mapping.
My Import scripts drop a full log of all entity values before running the $entity.Create ... $entity.Commit loop. There are no duplicate sAMAccountNames.
Note that when I did the IdB database upgrade in Test I removed the final lines from the script as told to do here. I don't seem to have had any problems with this in Dev, but thought it worth mentioning.
Change detection engine import all items failed. Change detection engine import all items for connector PowerShell HomeFolder NMI failed with reason An error occurred while evaluating a task on a worker thread. See the inner exception details for information.. Duration: 00:01:37.5326976 Error details: Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentException: An item with the same key has already been added. at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector) at Unify.Product.IdentityBroker.Repository.EntityLinqQueryConverterUtilitiesBase`4.GetCollectionKeyData(TEntityKey key, EntityDataContext sourceContext) at Unify.Product.IdentityBroker.Repository.EntitySingleValueDataUtilityBase`2.CreateEntityValue(TEntityKey key, IValue value, IEntityCollectionKeyUtility`1 collectionKeyUtility, EntityDataSet set, __EntityInsertRow row, EntityDataContext sourceContext) at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.ConvertEntityValueToDataValue(KeyValuePair`2 entityValueAndKey, __EntityInsertRow row, EntityDataSet entityDataSet, EntityDataContext sourceContext) at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.<>c__DisplayClass31_0.<convertitemtovalues>b__0(KeyValuePair`2 entityValueAndKey) at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext() at System.Linq.Enumerable.<selectmanyiterator>d__16`2.MoveNext() at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor) at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.InsertItems(ISet`1 addedItems, EntityDataContext sourceContext, SqlConnection connection) at Unify.Framework.Data.LinqContextConversionBase`4.SubmitChanges() at Unify.Product.IdentityBroker.SaveChangedEntitiesTransformationUnit.Transform(IDictionaryTwoPassDifferenceReport`4 input) at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys) at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_0.<performchangedetection>b__0(IEnumerable`1 page) at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate() --- End of inner exception stack trace --- at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException() at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit() at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities) at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess() at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase() at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run() at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.<run>b__0() at Unify.Framework.<span class="redactor-selection-marker" id="selection-marker-1"></span>AsynchronousJobExecutor.PerformJobCallback(Object state) </run></performchangedetection></selectmanyiterator></convertitemtovalues>
Answer
I think it has something to do with duplicate schema names in the database. How do I fix that? If I run this:
select ck.[CollectionKeyId], ck.Caption from [dbo].[CollectionKey] ck join ( select [Caption] from [dbo].[CollectionKey] group by Caption having count(Caption) > 1 ) dup on ck.Caption = dup.Caption order by ck.Caption ascI get this:
CollectionKeyId Caption
--------------- -------
203 HomePostcode
238 HomePostCode
114 OrganisationUnitDescription
100 OrganisationUnitDescription
99 OrganisationUnitLevel
110 OrganisationUnitLevel
97 OrganisationUnitNumber
109 OrganisationUnitNumber
98 OrganisationUnitStatus
115 OrganisationUnitStatus
95 OrgUnit1ID
101 OrgUnit1ID
102 OrgUnit2ID
111 OrgUnit2ID
103 OrgUnit3ID
112 OrgUnit3ID
104 OrgUnit5ID
113 OrgUnit5ID
105 OrgUnit6ID
96 OrgUnit6ID
106 OrgUnit7ID
116 OrgUnit7ID
194 OrgUnit7Name
192 OrgUnit7Name
195 OrgUnit7Ref
191 OrgUnit7Ref
107 OrgUnit8ID
117 OrgUnit8ID
108 OrgUnit9ID
118 OrgUnit9ID
128 OSUserId
50 OsUserId
216 Postcode
243 PostCode
126 UserId
224 UserID
I noticed that the value in OsUserID has disappeared in the adapter in Test and there's now also a reflection error for that adapater. The connector still has the value, that particular attribute is not used in any other connector, and is not renamed in the adapter.
Hi Carol,
You're correct. Duplicate captions are the cause which probably originated in the previous version. If you haven't already, try clearing the CollectionKey table. This will cause IdB to re-populate it, hopefully with no duplicates. Let me know if you notices v5.2 generating duplicates and continuing to raise this exception.
Actually, clearing won't work due to existing entities relying on that collection key id. You will need to perform a selective delete where the CollectionKeyId is not references in the EntityValue or EntityValueOriginTables. For example:
DELETE FROM CollectionKey
WHERE CollectionKeyId NOT IN (SELECT CollectionKeyId FROM EntityValue)
AND (CollectionKeyId NOT IN (SELECT CollectionKeyId FROM EntityValueOrigin)
OR CollectionKeyId NOT IN (SELECT SourceCollectionKeyId FROM EntityValueOrigin))
Once done, run your query from above again and there will hopefully the only duplicates will be have different casing.
For future reference, this issue is caused by entries in the CollectionKey table with the same Caption field value. The duplicate captions, produced by a defect in Identity Broker v5.1, cause exceptions to be thrown in several areas of the application after performing an upgrade to Identity Broker v5.2 which assume these values to be unique.
The simplest solution for this issue would be running the database clear script found in the <InstallDir>/Database directory. If this is not possible or desirable, attempt to run the script I provided below which clears the CollectionKey table of all unused entries and may resolve this issue. If the issue persists at this point a script or tool can be provided suitable to the specific environment to more directly correct the database state.
Customer support service by UserEcho
For future reference, this issue is caused by entries in the CollectionKey table with the same Caption field value. The duplicate captions, produced by a defect in Identity Broker v5.1, cause exceptions to be thrown in several areas of the application after performing an upgrade to Identity Broker v5.2 which assume these values to be unique.
The simplest solution for this issue would be running the database clear script found in the <InstallDir>/Database directory. If this is not possible or desirable, attempt to run the script I provided below which clears the CollectionKey table of all unused entries and may resolve this issue. If the issue persists at this point a script or tool can be provided suitable to the specific environment to more directly correct the database state.