Locker entity not deprovisioned when adapter entity join field values change

A possible workaround to avoid this problem is to use all of the Join criteria fields in the adapter's DN.  This worked for me in one case, but not in another, so no guarantees.

Matt believes that UNIFYBroker re-evaluates a link's Join criteria before any provision, deprovisioning or mappings take place.  So if an adapter's link join fields change, the join is somehow lost and the old locker entity is abandoned.

However, this behaviour doesn't occur if the adapter's DN changes as well.  I can only intuit that there must be a check that takes place before the Join criteria is evaluated, that notices that the entity with the old DN no longer exists and triggers the deprovision of the locker entity

Here's an example.  The locker before:

The third field (which is part of the Key for the underlying connector and now a part of the DN for the adapter) is changed from 01 to 33.  The following locker updates occur:

The entity IDs confirm that the old locker entity has been deleted and a new one created.

When the adapter's DN didn't include the link's join criteria field, the behaviour was that although a new locker entity was created the old one was left behind as well.

Also, this same issue occurs for outbound deprovisioning as well (i.e. duplicate adapter entities get created when join criteria field values change in a locker entity for a link configured with outbound provisioning and deprovisioning).

To replicate this problem for outgoing deprovisioning:

• Create SQL table with one varchar field and one record.
• Create SQL connector for the table, called ‘incoming’. Set the field to Key, Required. Import All. Record the connector entity ID.
• Create adapter ‘incoming’. DN template whatever you like, doesn’t matter. Generate Changes.
• Create locker ‘demo’ with one field.
• Create CSV connector ‘outgoing’ for a file with one line with one field name. Set the field to Key, Required. Import All.
• Create adapter ‘outgoing’ with DN template CN=@IdbID. Generate Changes.
• Create link ‘incoming’ from connector ‘incoming’ to locker ‘demo’ with join on the one field and adapter to locker mapping for the one field. Enable incoming provisioning and deprovisioning. Run Baseline Sync. Record the locker entity ID.
• Create link ‘outgoing’ from locker ‘demo’ to connector ‘outgoing’ with join on the one field and locker to adapter mapping for the one field. Enable outgoing provisioning and deprovisioning. Run Baseline Sync. Record the adapter entity ID.
• Change the value of the one record in the SQL table.
• Run Import All on connector ‘incoming’. Note that the connector entity ID changes.
• Run Changes Synchronisation on the ‘incoming’ link. Note the locker entity ID changes.
• Run Changes Synchronisation on the ‘outgoing’ link. Note that a new adapter entity is created, but the old one remains as well.

To replicate this problem for incoming deprovisioning:

• Create SQL table with two varchar fields and one record. One field will be used as a Key on the connector, causing the connector entity ID is to be retained.
• Create SQL connector for the new table, called ‘incoming2’. Set the first field to Key, Required. Import All. Record the connector entity ID.
• Create adapter ‘incoming2’. DN template whatever you like, doesn’t matter. Generate Changes.
• Create locker ‘demo2’ with two fields.
• Create link ‘incoming2’ from connector ‘incoming2’ to lock ‘demo2’ with join the second field (secondfieldname) and adapter to locker mapping for both fields. Enable incoming provisioning and deprovisioning. Run Baseline Sync. Record the locker entity ID.
• Change the value of the second field of the one record in the SQL table.
• Run Import All on connector ‘incoming2’. Note the connect entity ID does not change.
• Run Changes Synchronisation on the ‘incoming2’ link. Note that a new locker entity is created, but the old one remains as well.

Notes:

For incoming, the problem always happens when the connector entity ID (which is identical to the adapter entity ID) remains unchanged.  This doesn't happen when the source connector has a Key which matches the (incoming) link's join criteria fields, since the change of those fields causes the connector entity ID to change.

For outgoing, the problem always happens, because the locker entity ID doesn't change for new values of the (outgoing) link's join criteria fields.

The above replication only works in one of my customer environments, and not in a testbed environment Matt gave me access to.  Further investigation is under way.

Hi Adrian

A patch with the changes we discussed yesterday has been deployed into the test Connect instance. You will find that links now have a new configuration option Join Strategy. It should be Simple Join Resolution by default, which is the existing behaviour. Change it to Connection-aware Join Resolution to enable the new behaviour. You should clear the locker of all entities after switching join strategy, especially going from simple -> connection-aware.

As mentioned yesterday, the new join strategy is more strict than the existing one, and will abort the sync with an error message if it encounters an invalid state that it cannot recover from. The error message should hopefully be self-explanatory and include the IDs of any relevant entities.

Let me know how it goes, if you have any issues, or if you have any other feedback.

Thanks Beau, that's awesome!  I'll test it out as soon as I can.

I have retested this today and the results are very good.  The UI problem with not displaying the Join Strategy was no longer evident.

For both Baseline Sync and Changes Sync when the Join Strategy is Connection-aware Join Resolution and the Join criteria did not match the connector keys the data outcomes were perfect: entities were retained as expected and updated correctly with no duplicate records created.

One warning was recorded for the outgoing Baseline Sync, but this did not affect the data veracity at all:

Baseline synchronization encountered 1 incomplete sync changes. Ensure that either the field/s used in the join rules are correctly mapped or, if this link is not responsible for provisioning, the joining entities already exist. If adapter entities are to be provisioned, ensure that all adapter schema fields marked 'Required' are being mapped to with valid data.

I also tested with Join criteria matching the connector key and that still works as expected.

(next up, testing deprovisioning - the original issue for this ticket)

Here are the result for deprovisioning:

Tested Baseline and Changes for cases of Join criteria equal to connector Keys and not, always with Join Strategy set to Connection-aware.  In all cases the date updates were correct (i.e. target entities were deleted).

Two minor issues were evident:

1. On the outgoing Baseline Sync for the deprovision operation the following warning was logged:

Baseline synchronization encountered 1 incomplete sync changes. Ensure that either the field/s used in the join rules are correctly mapped or, if this link is not responsible for provisioning, the joining entities already exist. If adapter entities are to be provisioned, ensure that all adapter schema fields marked 'Required' are being mapped to with valid data.
20210212,01:07:53,UNIFYBroker,ProvisioningExecutor,Warning,The following entities [Count:1] for the link out (5972707c-398a-434e-936a-449b23663d1f) are missing the field used for the join criteria: 69b7ee05-cae8-4ef1-97af-e14d086744b8: [ id ],Normal

2. After the outgoing sync the 'Pending Incoming Sync Changes' increased (as usual) but then neither a Baseline Sync nor a Changes Sync were able to clear the count like they normally would.  Clicking on the counter to show the pending sync changes showed 'No data available in table'.  This happened for both cases (i.e. Join criteria equal or not equal to the connector Keys).