Entity not updating after Foreign Multivalue Group relationship connector entity change

Hi Adrian

What is the extent of this issue? Does it occur for all membership changes for any group or just the one group you specified? Is it repeatable or did it only occur once?

I'll see if I can replicate it.

The customer is reporting that this is continuing to happen, for all changes they make.

I tested this in the DEV environment and it works just fine.  The issue is only occurring in the TEST environment.

I've identified 2 separate defects, both related to the updating of adapter entities through a FMG transformation involving empty groups.

Firstly, when an empty group is updated to include a new member an exception is throw, which interrupts the change detection process, preventing further. The exception is logged, and looking at the current days logs (UnifyLog20210120.csv) I see one instance of it.

20210120,08:45:11,UNIFYBroker,Change detection engine,Error,"Changes register item processing on failed.
Changes register item processing on connector AD Groups failed with reason Value cannot be null.
Parameter name: collection. Duration: 00:00:00.0460000
Error details:
System.ArgumentNullException: Value cannot be null.
Parameter name: collection
   at System.Collections.Generic.HashSet`1..ctor(IEnumerable`1 collection, IEqualityComparer`1 comparer)
   at Unify.Product.IdentityBroker.MultiRelationalTransformationContribution.GetChangedMultiValues(IEntityPair entityPair, Boolean relevantFieldsChanged)
   at System.Linq.Enumerable.<selectmanyiterator>d__23`3.MoveNext()
   ...

Secondly, when removing the last member from a group, the change detection looses track of the relationship between relation connector entity and the adapter entity. This is a logical error, so there's no exception and will only affect the entities directly involved.

The first defect is both the most impactful and simplest to fix so I'll work on getting that patched first, then investigate the second further.

Thanks Beau.

The original problem occurred when a second value was added to a multi-valued field which already had an existing value, so it doesn't sound like the either of the two cases you have identified (assuming you mean "empty multi-valued field" where you wrote "empty group"?  The multi-valued field that is changing is in the "AD Entitlement Control Groups" connector - the "AD Groups" adapter is the one which references it via a FMG transform.

I can trigger the problems at will, so I'll do so and re-watch the error log to see if anything is written.

Yeah, by "empty group" I mean a entity in the groups connector with an empty membership multi value field.

Also, just to be clear, the part change detection process which contains the above defects only runs when there are actual changes. Meaning if a group membership change has already been imported into the connector reimporting won't do anything, ie only when going from [] to [ user01 ].

I added SSAS-CDSPROD-Sales-Admin to the Information Technology control group (which already has some values in the multi-valued field) and it worked correctly.

I added SSAS-CDSPROD-ISManager to the BI control group (the one I saw the problem on yesterday, already has other values) and it also worked.

The control group the customer reported not working this morning is RolePush, and it currently has two member groups, neither of which have worked correctly. I added SSAS-CDSDEV-CorporateFinance to it, and that group worked, but the two groups that didn't work previously (Role-IT-Zendesk and Role-IT-Data) still haven't been processed correctly yet.

So it would appear the failure isn't manifesting today, but the past data is still there with the problem and it isn't repairing itself (unless I manually run a Generate Changes).

Hi @Beau - what's the next step for us here?  If we don't know what caused it then how can we be sure it won't happen again?  And if it does then what can we do to work out the root cause and fix it?

While you say the issue was caused by added a second member to a group, the fact you couldn't reproduce this in isolation, and that the expected exception was seen in the logs suggests that the root cause was indeed the first bug I detailed above. As I said, this bug being triggered for an membership change would prevent the processing of any subsequent changes that occurred in the same import.

David has deployed a patch which fixes these issues. Please run the following process to confirm:

1. Ensure the users and groups connectors are both fully imported, and manually generate changes on the adapter to ensure it is in the correct state.
2. Run the following steps on one or more groups. If possible, include the users and groups that were involved in the initial discovery of the issue. After each step, import the group connector and observe that the changes are reflected into the adapter. Also keep an eye on the logs for any error that may appear.
   1. Remove all members from the group
   2. Add one or more member to the empty group
   3. Add another one or more member to the group
   4. Remove all but 1 member from the group
   5. Remove the last member of the group

Thanks Beau, I following the confirmation process - no problems encountered.

Let's hope you nailed it :)

Thank you!

Unfortunately this problem has just happened again for the customer.  Matt has asked me to contact Beau directly via Teams to escalate.

Beau has investigated and has advised he will provide another patch.

Patch has been applied in the customer environment and Adrian will test.

I confirm that I have tested and this latest patch appears to have solved this issue.

Thanks Beau!

Hi all,
This bug appears to still be happening in the Netwealth DEV environment, and has caused SIT to be failed by Adi.
Please could you re-open this and investigate further?
Thanks,
A.

Hi Adrian,

Have you validated with the steps Beau published above to recreate?

Will pick this up next week

Hi Matthew & Beau

I have been unable to replicate this problem using the steps written up above by Beau, or by trying a number of other similar data actions that I hoped might be relevant.

I don't know the initial state or steps undertaken by Adi which caused the problem to manifest.

As requested by Matt (inTeams chat) I have sought advice from Adi regarding the steps he followed to cause this problem and tried to replicate it myself.  I could not replicate this issue (although I did encounter two other issues are have raised separate tickets for them: https://voice.unifysolutions.net/en/communities/6/topics/4216-ad-connector-import-changes-doesnt-pick-up-changes-to-multivalued-attribute-import-all-does-for-ad and  https://voice.unifysolutions.net/en/communities/6/topics/4217-connection-aware-join-not-persisting-for-outgoing-link-when-join-criteria-field-value-changes-new )

So we are back to square one - this issue has been observed but is not readily reproducable.