Duplicates from Aurion

Could you please confirm that the format looks like this?

<DocumentRoot>
  <!--data-->
</DocumentRoot>
  <!--data-->
</DocumentRoot>

It might be possible to ignore the second set of data (if it's not currently throwing an xml based exception I'm a little shocked as it's not a valid document...), however, if we start ignoring data issues we'll be hiding the fact that there is a bug in Aurion and possibly let performance (or worse) issues go unfixed. There's also the question about whether the closing DocumentRoot element will always be at the end of a set of data instead of the middle. In this case it might look like we're adding resilience when it's actually making it more susceptible to problems.

Yes that's how it looks, and yes I get all the concerns - it's just causing me a problem right now because it's the Person report doing it this time and I'm not getting any new starters through. It's in Test but still a problem.

So far every one I've looked at has followed the same pattern, and I've seen at least three different reports do this, then come good. At this point I don't know if the issue is just on their test Aurion server, or if I'll see it higher up (hnce my question on yammer wondering if anyone else had seen this).

The error IdB is reporting is the duplicate key one - which is interesting because the XML is mal-formed.

Out of interest, could you also add the full error details here?

Change detection engine import all items failed.

    Change detection engine import all items for connector Aurion Person Connector
failed with reason An error occurred while evaluating a task on a worker
thread. See the inner exception details for information.. Duration:
00:14:17.1028573
    
    Error details:
    Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a
task on a worker thread. See the inner exception details for information.
---> System.ArgumentException: The key <null> has been duplicated.
    at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue`1 arg1)
    at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)
    at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)
    at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 originalEntities, IHashSet`1 seenKeys)    
    at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
    at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_0.<PerformChangeDetection>b__0(IEnumerable`1 page)
    at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()

    --- End of inner exception stack trace ---
    at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException()
    at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.WaitForAvailableThread()
    at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
    at Unify.Framework.Visitor.VisitEvaluateOnThreadPool[T](IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
    at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
    at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
    at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
    at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
    at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.<Run>b__0()
    at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)

Just make it clear - there are no NULL keys - I've checked the raw XML returned using soapUI.

That error isn't related to the bad xml document (although that may also be an issue). For the duplicate null entry see http://voice.unifysolutions.net/topics/2821-connector-failed-with-reason-the-key-null-has-been-duplicated/

Yes I've had that plenty of times for that reason - but this time the schema hasn't changed in IdB or in Aurion since the report last worked, so I'm pretty sure it relates to the duplicated data.

Hold that - I could be wrong - will double-check. They refreshed Aurion from Prod a couple of days ago, and all my reports were loaded into Prod first - better make sure they didn't load an old version.

Could you attach your connector config file?

Without the xml output it's going to be tricky to diagnose, because I can't get it to ignore the duplicate close tag. Are there any entries that are missing the key field?

Now the missing key field has been added back into the report it's working properly with IdB. I was hoping it would still be sending the duplicate data when run through soapUI, just to see what the new IdB error would be - but the duplicates are no longer there. So while my import is no working I still don't know what is causing Aurion to sometimes do this.

If you can find a sample xml file please let me know as I'm unable to reproduce it ignoring the duplicate close tag. Thanks.