I put through an update from Azure via SCIM which had two attributes changing at the same time, and the following error appeared in the UNIFYBroker log:

20220422,05:36:27,UNIFYBroker,SCIMGateway,Error,"Error during SCIM operation: System.InvalidOperationException: Sequence contains more than one element
at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
at Unify.Product.IdentityBroker.SCIMProvider.Patch(IAdapterEntity adapterEntity, PatchRequest2 patch, ISCIMGatewayMapping mappings, IValueAdapter`2 valueAdapter, IEntitySchema schema)
at Unify.Product.IdentityBroker.SCIMProvider.d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.SystemForCrossDomainIdentityManagement.ProviderBase.d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.SystemForCrossDomainIdentityManagement.ProviderAdapterTemplate`1.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.SystemForCrossDomainIdentityManagement.ControllerTemplate`1.d__5.MoveNext()",Normal

Here is the corresponding update from the Azure side:

As a possible complication for this ticket, the user involved previously didn't have a value for their employeeNumber field (which is a required/key field of the underlying connector) and that caused this error:

20220422,05:34:08,UNIFYBroker,An exception has occured in the AdapterEntityChangeDetectorCollator. Please see the exception for details,Error,"Npgsql:
System.InvalidOperationException: Parameter 'key1value0' must have its value set
at Npgsql.NpgsqlParameter.ResolveHandler(ConnectorTypeMapper typeMapper)
at Npgsql.NpgsqlParameter.Bind(ConnectorTypeMapper typeMapper)
at Npgsql.NpgsqlCommand.ValidateParameters()
at Npgsql.NpgsqlCommand.d__100.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlCommand.d__92.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlCommand.ExecuteNonQuery()
at Unify.Product.IdentityBroker.MultiKeyValueQueryExecutor`2.PopulateTempTable(NpgsqlConnection connection, NpgsqlTransaction transaction, IDictionary`2 tempKeyLookup, ValueToObjectAdapter adapter, IEnumerable`1 multiKeyValues, String tempTableName)
at Unify.Product.IdentityBroker.MultiKeyValueQueryExecutor`2.Execute(NpgsqlConnection connection, ValueToObjectAdapter adapter, Guid partitionId, IEnumerable`1 multiKeyValues, String tempTableName, String entityValueFieldName, String entityTableName, String partitionIdName, Func`2 readData)
at Unify.Product.IdentityBroker.KnownEntityPartitionPostgreSqlUpdatableContextBase`3.GetEntities(Guid partitionId, IEnumerable`1 keys)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.<>c__DisplayClass1_0.b__0(MultiKeyValue`1[] multiKeyValues)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, Func`2 retrieveEntities, Guid connectorId, IEnumerable`1 wellKnownEntities)
at Unify.Product.IdentityBroker.EntityRepositoryExtensions.ConvertConnectorEntities(IEnumerable`1 connectorEntities, IMultiKey`1 schemaKey, IKnownEntityContextBase`3 context, Guid connectorId, IEnumerable`1 wellKnownEntities)
at Unify.Product.IdentityBroker.EntityChangeDetector.ProcessConnectorChangedEntities(Guid connectorId, IEnumerable`1 connectorEntities, IEnumerable`1 wellKnownItems)
at Unify.Product.IdentityBroker.AdapterEntityChangeDetectorCollator.DetectChanges(KeyValuePair`2 connectorEntities)
at Unify.Framework.Visitor.<>c__DisplayClass0_0`1.b__0(T item, Int32 index)
at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
at Unify.Product.IdentityBroker.AdapterEntityChangeDetectorCollator.Run()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal

Could you please investigate and fix the update error, and also consider improving the behaviour when a connector's required field is missing in a SCIM update?

For the purposes of my demo next week I can make sure this 'bad data' doesn't occur, and so the only dependency for this is SIT/UAT meaning that it's a medium priority.

I have a SCIM gateway configured in UNIFYConnect and Azure is able to use it to Add users just fine, but when it attempts an update this error is shown:

There's nothing written into the UNIFYBroker log when this happens.

Could you please investigate and advise how to make updates work?

The strangest thing ever... For this adapter:

When I generate changes to update the DNs (used to be DN=@IdBID), I see DNs assigned with different segment orders like this:

The two digit CN is DepartmentID.  The date CN is StartDate.  The four digit CN is PositionID.  The three letter CN is LocationID.  The six digit CN is EmployeeID.

I tried clearing precalculated entities and re-running Generate Changes, but the same thing happens with the newly generated entities - the segments are always in the same (wrong) order for each entity.

This is happening in a non-Production environment, and the DN segment order for each entity appears to be persistent, so this is a very minor (but very strange) issue.

My customer's adapter has two entities for every every connector entity.  I am not aware of what might have caused this, other than the service outage that was resolved yesterday (unresponsive UI and no data was flowing through the system).

Could you please advise what caused there to be two adapter entities for each connector entity?

Azure's SCIM implementation when creating a user who has a manager follows this approach:

1. Create the new SCIM user (without the manager reference)
2. Make sure the new user has been created
3. Update the new user's manager field

It seems the reason for this approach is because Azure wants to create all the users first, before it tries to add references to them via the manager field (which makes sense from a referential integrity perspective).

However, when using the UNIFYBroker SCIM gateway, step 2 is failing and Azure can't find the newly created user.  Azure first tries to get the new user object by ID, and then by username lookup, but neither works.  See the attached packet trace "SCIM User with manager creation.pcapng" for details.  The connector/adapter entity created has the same entity ID that was returned to Azure (9a9978e0-6179-42b5-8218-2173d7b6c0e5 in the packet trace) and the username field set correctly (adrian.corston@unifysolutions.net).

A later attempt by Azure to look up that entity by username (package trace "SCIM User lookup.pcapng") is successful (that lookup was followed by a PATCH which failed, but that is due to a different issue so please ignore that part of the trace).

SCIM User with manager creation.pcapng
SCIM User lookup.pcapng

This github ticket has information about the Azure SCIM pattern for setting manager: https://github.com/MicrosoftDocs/azure-docs/issues/11784 (comment from asmalser-msft)

Hi guys,

I am using the SCIM gateway in v5.3.2 and the department and manager attributes don't appear to work (i.e. the values sent by SCIM are not populated into the UNIFYBroker adapter fields).

Here's my configuration in Azure:

It seems possible that the problem might be related to the fact those two attributes are extension attributes and not part of the SCIM core user schema: https://datatracker.ietf.org/doc/html/rfc7643#section-4.1

As a separate request could you please add support for the givenName and familyName core user attributes?

I am setting up a SCIM gateway in UNIFYBroker, and it is returning a 401 (Unauthorized) error.

I have checked the Bearer token in the request header matches the Audience configuration in UNIFYBroker.

Do you have any suggestions what might be going wrong?

Obviously once this is working I will change the secret token shown above.

Given this config:

and target connector data like this:

I am seeing reflection fail (no entities processed) with this error logged:

20220121,07:27:22,UNIFYBroker,Adapter,Error,"Request to reflect change entities of the adapter.
Request to reflect change entities of the SPOL Azure-mastered AD Groups (928e6c08-48d0-48da-8330-08ce0df4ef55) adapter errored with message: One or more errors occurred.. Duration: 00:00:01.4730403
Error details:
System.AggregateException: One or more errors occurred. ---> Unify.Framework.Collections.GroupedNameValueCollectionMissingFieldException: The entity does not contain a value for the onPremisesUserPrincipalName field.
   at Unify.Product.IdentityBroker.EntityBase`3.GetValueEntry(TKey key)
   at Unify.Product.IdentityBroker.RelationshipEntityProcessor.d__1.MoveNext()
   at System.Linq.Enumerable.d__17`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.SimpleValueMultivalueGroupTransformation.TransformItem(IEntity leftSideEntity, ILookup`2 rightSideLookup, IEntitySchemaFieldDefinition primaryKeyField)
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
   at System.Linq.Parallel.PipelineSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
   at System.Linq.Parallel.AsynchronousChannelMergeEnumerator`1.MoveNextSlowPath()
   at System.Linq.Parallel.QueryOpeningEnumerator`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at System.Linq.Enumerable.Aggregate[TSource,TAccumulate](IEnumerable`1 source, TAccumulate seed, Func`3 func)
   at Unify.Product.IdentityBroker.Adapter.ReflectChangePage(IChangesRegisterKey[] changedPage, IEntityPartitionContext connectorContext, IAdapterEntityPartitionUpdatableContext adapterContext, ITransformation transformation, DuplicateDnDetector duplicateDnDetector)
   at Unify.Product.IdentityBroker.Adapter.ReflectChangesInner()
   at Unify.Product.IdentityBroker.Adapter.ReflectChanges()
   at Unify.Product.IdentityBroker.AdapterAuditingDecorator.ReflectChanges()
   at Unify.Product.IdentityBroker.AdapterNotifierDecorator.ReflectChanges()
   at Unify.Product.IdentityBroker.ReflectAdapterOnChangeDueJob.RunOnAdapter(IOperationalAdapter adapter)
---> (Inner Exception #0) Unify.Framework.Collections.GroupedNameValueCollectionMissingFieldException: The entity does not contain a value for the onPremisesUserPrincipalName field.
   at Unify.Product.IdentityBroker.EntityBase`3.GetValueEntry(TKey key)
   at Unify.Product.IdentityBroker.RelationshipEntityProcessor.d__1.MoveNext()
   at System.Linq.Enumerable.d__17`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.SimpleValueMultivalueGroupTransformation.TransformItem(IEntity leftSideEntity, ILookup`2 rightSideLookup, IEntitySchemaFieldDefinition primaryKeyField)
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
   at System.Linq.Parallel.PipelineSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
   at System.Threading.Tasks.Task.Execute()<---
---> (Inner Exception #1) Unify.Framework.Collections.GroupedNameValueCollectionMissingFieldException: The entity does not contain a value for the onPremisesUserPrincipalName field.
   at Unify.Product.IdentityBroker.EntityBase`3.GetValueEntry(TKey key)
   at Unify.Product.IdentityBroker.RelationshipEntityProcessor.d__1.MoveNext()
   at System.Linq.Enumerable.d__17`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.SimpleValueMultivalueGroupTransformation.TransformItem(IEntity leftSideEntity, ILookup`2 rightSideLookup, IEntitySchemaFieldDefinition primaryKeyField)
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
   at System.Linq.Parallel.PipelineSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
   at System.Threading.Tasks.Task.Execute()<---
",Normal

In terms of correct functionality, since onPremisesUserPrincipalName is the Reference field I would expect a join to a target connector entity where the onPremisesUserPrincipalName value is NULL to be ignored (i.e. a NULL onPremisesUserPrincipalName value wouldn't be populated into MemberADUPNs).

Note: this ticket might be related to https://voice.unifysolutions.net/en/communities/6/topics/4082-foreign-multivalued-group-transformation-reports-value-cannot-be-null-for-multivalued-attribute but that one is a different transform and looks like a subtly different manifestation, so maybe not.

In my customer's UAT and PROD environments an adapter is retaining old entities when a connector import all operation completes with less entities (i.e. some entities have been deleted). This happens every time the import all operation is run.  It seems some updated/new entities may not be being processed either, but that's not as clear.  What is clear is that the adapter entity count increases to a number greater than the connector's entity after the import all.  There are no corresponding errors in the log.  Generate changes does not fix the problem, but deleting the adapter entities and then running generate changes does.

What information would you like to help debug this?

Version is v5.3.3 Revision #0
The connector is a PowerShell one

This error is being written to the UNIFYBroker logs in multiple UNIFYConnect environments.  I deleted all transforms that referred to those fields, removed references to the fields from links, and then recreated them all, but it didn't stop the error from appearing.