An item with the same key has already been added
I have just upgraded IdB in TEST to 5.2, and migrated in the Connector and Adapter files from Dev. Dev was already on 5.2 and all connectors are working.
In TEST all of the new Connectors (names "PowerShell HomeFolder*" and "PowerShell MemberOf*") are failing with the error below. The "Powershell Exchange*" connectors work fine (though they already pre-existing in IdB 5.1 before I upgraded to 5.2).
The Connector config file is the same as the one I sent with the previous question. While the error looks very similar to that one it can't be the same - that was a duplicate schema mapping in the connector config, but Powershell connectors don't have schema mapping.
My Import scripts drop a full log of all entity values before running the $entity.Create ... $entity.Commit loop. There are no duplicate sAMAccountNames.
Note that when I did the IdB database upgrade in Test I removed the final lines from the script as told to do here. I don't seem to have had any problems with this in Dev, but thought it worth mentioning.
Change detection engine import all items failed. Change detection engine import all items for connector PowerShell HomeFolder NMI failed with reason An error occurred while evaluating a task on a worker thread. See the inner exception details for information.. Duration: 00:01:37.5326976 Error details: Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentException: An item with the same key has already been added. at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector) at Unify.Product.IdentityBroker.Repository.EntityLinqQueryConverterUtilitiesBase`4.GetCollectionKeyData(TEntityKey key, EntityDataContext sourceContext) at Unify.Product.IdentityBroker.Repository.EntitySingleValueDataUtilityBase`2.CreateEntityValue(TEntityKey key, IValue value, IEntityCollectionKeyUtility`1 collectionKeyUtility, EntityDataSet set, __EntityInsertRow row, EntityDataContext sourceContext) at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.ConvertEntityValueToDataValue(KeyValuePair`2 entityValueAndKey, __EntityInsertRow row, EntityDataSet entityDataSet, EntityDataContext sourceContext) at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.<>c__DisplayClass31_0.<convertitemtovalues>b__0(KeyValuePair`2 entityValueAndKey) at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext() at System.Linq.Enumerable.<selectmanyiterator>d__16`2.MoveNext() at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor) at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.InsertItems(ISet`1 addedItems, EntityDataContext sourceContext, SqlConnection connection) at Unify.Framework.Data.LinqContextConversionBase`4.SubmitChanges() at Unify.Product.IdentityBroker.SaveChangedEntitiesTransformationUnit.Transform(IDictionaryTwoPassDifferenceReport`4 input) at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys) at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_0.<performchangedetection>b__0(IEnumerable`1 page) at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate() --- End of inner exception stack trace --- at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException() at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit() at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities) at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess() at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase() at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run() at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.<run>b__0() at Unify.Framework.<span class="redactor-selection-marker" id="selection-marker-1"></span>AsynchronousJobExecutor.PerformJobCallback(Object state) </run></performchangedetection></selectmanyiterator></convertitemtovalues>
Answer
I think it has something to do with duplicate schema names in the database. How do I fix that? If I run this:
select ck.[CollectionKeyId], ck.Caption from [dbo].[CollectionKey] ck join ( select [Caption] from [dbo].[CollectionKey] group by Caption having count(Caption) > 1 ) dup on ck.Caption = dup.Caption order by ck.Caption ascI get this:
CollectionKeyId Caption --------------- ------- 203 HomePostcode 238 HomePostCode 114 OrganisationUnitDescription 100 OrganisationUnitDescription 99 OrganisationUnitLevel 110 OrganisationUnitLevel 97 OrganisationUnitNumber 109 OrganisationUnitNumber 98 OrganisationUnitStatus 115 OrganisationUnitStatus 95 OrgUnit1ID 101 OrgUnit1ID 102 OrgUnit2ID 111 OrgUnit2ID 103 OrgUnit3ID 112 OrgUnit3ID 104 OrgUnit5ID 113 OrgUnit5ID 105 OrgUnit6ID 96 OrgUnit6ID 106 OrgUnit7ID 116 OrgUnit7ID 194 OrgUnit7Name 192 OrgUnit7Name 195 OrgUnit7Ref 191 OrgUnit7Ref 107 OrgUnit8ID 117 OrgUnit8ID 108 OrgUnit9ID 118 OrgUnit9ID 128 OSUserId 50 OsUserId 216 Postcode 243 PostCode 126 UserId 224 UserIDI noticed that the value in OsUserID has disappeared in the adapter in Test and there's now also a reflection error for that adapater. The connector still has the value, that particular attribute is not used in any other connector, and is not renamed in the adapter.
Hi Carol,
You're correct. Duplicate captions are the cause which probably originated in the previous version. If you haven't already, try clearing the CollectionKey table. This will cause IdB to re-populate it, hopefully with no duplicates. Let me know if you notices v5.2 generating duplicates and continuing to raise this exception.
Actually, clearing won't work due to existing entities relying on that collection key id. You will need to perform a selective delete where the CollectionKeyId is not references in the EntityValue or EntityValueOriginTables. For example:
DELETE FROM CollectionKey
WHERE CollectionKeyId NOT IN (SELECT CollectionKeyId FROM EntityValue)
AND (CollectionKeyId NOT IN (SELECT CollectionKeyId FROM EntityValueOrigin)
OR CollectionKeyId NOT IN (SELECT SourceCollectionKeyId FROM EntityValueOrigin))
Once done, run your query from above again and there will hopefully the only duplicates will be have different casing.
For future reference, this issue is caused by entries in the CollectionKey table with the same Caption field value. The duplicate captions, produced by a defect in Identity Broker v5.1, cause exceptions to be thrown in several areas of the application after performing an upgrade to Identity Broker v5.2 which assume these values to be unique.
The simplest solution for this issue would be running the database clear script found in the <InstallDir>/Database directory. If this is not possible or desirable, attempt to run the script I provided below which clears the CollectionKey table of all unused entries and may resolve this issue. If the issue persists at this point a script or tool can be provided suitable to the specific environment to more directly correct the database state.
Customer support service by UserEcho
For future reference, this issue is caused by entries in the CollectionKey table with the same Caption field value. The duplicate captions, produced by a defect in Identity Broker v5.1, cause exceptions to be thrown in several areas of the application after performing an upgrade to Identity Broker v5.2 which assume these values to be unique.
The simplest solution for this issue would be running the database clear script found in the <InstallDir>/Database directory. If this is not possible or desirable, attempt to run the script I provided below which clears the CollectionKey table of all unused entries and may resolve this issue. If the issue persists at this point a script or tool can be provided suitable to the specific environment to more directly correct the database state.