Given this config:

and target connector data like this:

I am seeing reflection fail (no entities processed) with this error logged:

20220121,07:27:22,UNIFYBroker,Adapter,Error,"Request to reflect change entities of the adapter.
Request to reflect change entities of the SPOL Azure-mastered AD Groups (928e6c08-48d0-48da-8330-08ce0df4ef55) adapter errored with message: One or more errors occurred.. Duration: 00:00:01.4730403
Error details:
System.AggregateException: One or more errors occurred. ---> Unify.Framework.Collections.GroupedNameValueCollectionMissingFieldException: The entity does not contain a value for the onPremisesUserPrincipalName field.
   at Unify.Product.IdentityBroker.EntityBase`3.GetValueEntry(TKey key)
   at Unify.Product.IdentityBroker.RelationshipEntityProcessor.d__1.MoveNext()
   at System.Linq.Enumerable.d__17`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.SimpleValueMultivalueGroupTransformation.TransformItem(IEntity leftSideEntity, ILookup`2 rightSideLookup, IEntitySchemaFieldDefinition primaryKeyField)
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
   at System.Linq.Parallel.PipelineSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
   at System.Linq.Parallel.AsynchronousChannelMergeEnumerator`1.MoveNextSlowPath()
   at System.Linq.Parallel.QueryOpeningEnumerator`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at System.Linq.Enumerable.Aggregate[TSource,TAccumulate](IEnumerable`1 source, TAccumulate seed, Func`3 func)
   at Unify.Product.IdentityBroker.Adapter.ReflectChangePage(IChangesRegisterKey[] changedPage, IEntityPartitionContext connectorContext, IAdapterEntityPartitionUpdatableContext adapterContext, ITransformation transformation, DuplicateDnDetector duplicateDnDetector)
   at Unify.Product.IdentityBroker.Adapter.ReflectChangesInner()
   at Unify.Product.IdentityBroker.Adapter.ReflectChanges()
   at Unify.Product.IdentityBroker.AdapterAuditingDecorator.ReflectChanges()
   at Unify.Product.IdentityBroker.AdapterNotifierDecorator.ReflectChanges()
   at Unify.Product.IdentityBroker.ReflectAdapterOnChangeDueJob.RunOnAdapter(IOperationalAdapter adapter)
---> (Inner Exception #0) Unify.Framework.Collections.GroupedNameValueCollectionMissingFieldException: The entity does not contain a value for the onPremisesUserPrincipalName field.
   at Unify.Product.IdentityBroker.EntityBase`3.GetValueEntry(TKey key)
   at Unify.Product.IdentityBroker.RelationshipEntityProcessor.d__1.MoveNext()
   at System.Linq.Enumerable.d__17`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.SimpleValueMultivalueGroupTransformation.TransformItem(IEntity leftSideEntity, ILookup`2 rightSideLookup, IEntitySchemaFieldDefinition primaryKeyField)
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
   at System.Linq.Parallel.PipelineSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
   at System.Threading.Tasks.Task.Execute()<---
---> (Inner Exception #1) Unify.Framework.Collections.GroupedNameValueCollectionMissingFieldException: The entity does not contain a value for the onPremisesUserPrincipalName field.
   at Unify.Product.IdentityBroker.EntityBase`3.GetValueEntry(TKey key)
   at Unify.Product.IdentityBroker.RelationshipEntityProcessor.d__1.MoveNext()
   at System.Linq.Enumerable.d__17`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.SimpleValueMultivalueGroupTransformation.TransformItem(IEntity leftSideEntity, ILookup`2 rightSideLookup, IEntitySchemaFieldDefinition primaryKeyField)
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
   at System.Linq.Parallel.PipelineSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
   at System.Threading.Tasks.Task.Execute()<---
",Normal

In terms of correct functionality, since onPremisesUserPrincipalName is the Reference field I would expect a join to a target connector entity where the onPremisesUserPrincipalName value is NULL to be ignored (i.e. a NULL onPremisesUserPrincipalName value wouldn't be populated into MemberADUPNs).

Note: this ticket might be related to https://voice.unifysolutions.net/en/communities/6/topics/4082-foreign-multivalued-group-transformation-reports-value-cannot-be-null-for-multivalued-attribute but that one is a different transform and looks like a subtly different manifestation, so maybe not.

In my customer's UAT and PROD environments an adapter is retaining old entities when a connector import all operation completes with less entities (i.e. some entities have been deleted). This happens every time the import all operation is run.  It seems some updated/new entities may not be being processed either, but that's not as clear.  What is clear is that the adapter entity count increases to a number greater than the connector's entity after the import all.  There are no corresponding errors in the log.  Generate changes does not fix the problem, but deleting the adapter entities and then running generate changes does.

What information would you like to help debug this?

Version is v5.3.3 Revision #0
The connector is a PowerShell one

This error is being written to the UNIFYBroker logs in multiple UNIFYConnect environments.  I deleted all transforms that referred to those fields, removed references to the fields from links, and then recreated them all, but it didn't stop the error from appearing.

A few weeks ago all the connectors in a connector exclusion group inexplicably stopped running.  Scheduled Import All operations are running OK on connectors not in the connector exclusion group.  There does not appear to be any running Import All operations on any of the connectors in the group.

Hi,

Please let me know what could be the reason for 500 internal server error here.

Thank you

20210701,00:28:13,UNIFY Identity Broker,Connector,Warning,"Update entities to connector failed.

Update entities [Count:1] to connector TechOne SMS Student Management Users failed with reason The remote server returned an error: (500) Internal Server Error.. Duration: 00:00:02.1884079

Error details:

System.Management.Automation.RuntimeException: The remote server returned an error: (500) Internal Server Error. ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.

at System.Net.HttpWebRequest.GetResponse()

at CallSite.Target(Closure , CallSite , Object )

--- End of inner exception stack trace ---

at Unify.Product.IdentityBroker.PowerShellConnector.Execute(String script, IEnumerable`1 inputEntities, IEnumerable`1 inputKeys)

at Unify.Product.IdentityBroker.EventNotifierUpdatingConnectorDecorator.UpdateEntities(IEnumerable`1 entities)

at Unify.Product.IdentityBroker.Adapter.UpdateEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& connectorEntities)

at Unify.Product.IdentityBroker.Adapter.UpdateEntities(IEnumerable`1 entities)

at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.UpdateEntity(IAdapterEntity entityToSave)

at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.UpdateEntity(IAdapterEntity entityToSave)

at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportUpdate(IAdapter adapter, IAdapterEntitySaveChange pendingUpdate)

at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)

at SyncInvokeExportChanges(Object , Object[] , Object[] )

at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)

at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)

at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)

at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)

at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)",Normal

I have two MV Group transforms in an adapter:

The MemberUPNs field updated correctly, but MemberAzureIDs did not.  There are six values in MemberUPNs, but only two in MemberAzureIDs.  Manual validation confirms that the matching values in the second transform's connector are present and correct.  I do not know what the old value of MemberUPNs was.

I believe the change occurred on 30/6/21 some time in the hour or two prior to 8am, but unfortunately the corresponding log file has already been deleted.

Connector Processing page 1 for connector TechOne SMS Staff failed with reason The key CARLSSOS has been duplicated.. Duration: 00:00:00.9687755.

Error details:

System.ArgumentException: The key CARLSSOS has been duplicated.

at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue arg1)

at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)

at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IQueryable`1 sourceEntities, Guid connectorId, IEnumerable`1 originalEntities)

at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator)

at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass3.b__0(IEnumerable`1 page)

at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()",Normal

20210704,15:30:57,UNIFY Identity Broker,Connector Processor,Information,"Connector Processing started.

Connector Processing started for connector TechOne SMS Staff (page 2)",Normal

20210704,15:30:58,UNIFY Identity Broker,Connector Processor,Information,"Connector processing failed.

Connector Processing page 2 for connector TechOne SMS Staff failed with reason The key GOODMANN has been duplicated.. Duration: 00:00:01.4374637.

Error details:

System.ArgumentException: The key GOODMANN has been duplicated.

at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue arg1)

at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)

at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IQueryable`1 sourceEntities, Guid connectorId, IEnumerable`1 originalEntities)

at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator)

at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass3.b__0(IEnumerable`1 page)

at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()",Normal

20210704,15:31:23,UNIFY Identity Broker,Connector Processor,Information,"Connector Processing started.

Connector Processing started for connector TechOne SMS Staff (page 3)",Normal

20210704,15:31:25,UNIFY Identity Broker,Connector Processor,Information,"Connector processing failed.

Connector Processing page 3 for connector TechOne SMS Staff failed with reason The key ANDREWE has been duplicated.. Duration: 00:00:01.9218138.

Error details:

System.ArgumentException: The key ANDREWE has been duplicated.

at Unify.Product.IdentityBroker.EntityRepositoryExtensions.DuplicateKeyBase(MultiKeyValue arg1)

at Unify.Framework.Collections.EnumerableExtensions.ToDictionaryWithKeyClashError[TKey,TValue,TOriginal](IEnumerable`1 originalEnumerable, Func`2 keySelector, Func`2 valueSelector, Action`3 duplicateAction)

at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntitiesWithRepositoryEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IQueryable`1 sourceEntities, Guid connectorId, IEnumerable`1 originalEntities)

at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator)

at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass3.b__0(IEnumerable`1 page)

at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()",Normal

Hi,

We are using Unify identity Broker version v4.1.1 Revision #1 and one of the Connector is having below error  

Import All Run

Change detection engine import all items for connector Aurion Staging Staff Details Connector failed with reason The transaction has aborted.. Duration: 00:12:00.6306978

Based on my understanding, Connector is timing out if it's runs more than 10mins and I'm unable to find the set timeout threshold field limit in Identity broker. Can you please confirm where it could be set  or is there option to set the version we have.

Thank you

Below is from the logs

I have these adapter transforms set up:

Changes to the EmployeeNumbers attribute should flow through the following other attributes in this order:

1. MemberADUPNs (first MVG transform)
2. MemberAzureUPNs (PowerShell transform)
3. MemberAzureIDs (second MVG transform)

Sometimes when EmployeeNumbers values are updated I see MemberADUPNs and MemberAzureUPNs updating correctly, but MemberAzureIDs does not update with the new value like it should.

Q1. Is this because "PowerShell transforms break change detection"?  i.e. UNIFYBroker doesn't know that it needs to run the second MVG transform to update MemberAzureIDs, because it has no way to know that that MemberAzureUPNs may be updated by the PowerShell transform?

Q2. Could Beau's new "Register-Contribution" cmdlet be used in the PowerShell transform to make it so it does?

In ticket https://voice.unifysolutions.net/en/communities/6/topics/4238-time-offset-flag-didnt-re-evaluate-when-date-threshold-was-passed?redirect_to_reply=21889# Beau mentions that:

Add functionality to recreate deleted future-dated adapter changes to the UNIFYBroker UI (and API).

This should also be called automatically after a Clear Adapter Changes operation completes, because it is unlikely that there are any conceivable scenario where future-dated changes should not be restored, and without them UNIFYBroker does not operate sensibly (i.e. expected transforms silently fail to trigger when the configuration suggests they should).

Also, update the Clear Adapter Changes documentation in Voice to indicate that after running Clear Adapter Changes some transforms will no longer work correctly until Clear Precalculated Entities is run or all relevant date fields on all adapter entities are updated.