Investigate delta add behaviour with new adapter pre-populate functionality

Check to see if the changes register can be used during change detection, keeping in mind that the delta file shows whether the item in an add or an update.

Raising priority - has the potential to introduce major changes

Currently, delta imports trigger direct changes to the adapter context, ignorant of the reflect settings. The current work on reflection therefore does not yet have any bearing on adapter delta imports, only full imports. However, if the impact of reflection is to be realized on deltas, the reflection process itself will need some sort of handle on the changes register, and some mechanism for establishing attribute level changes.

Deltas will require changes having been calculated at the same time that the adapter context is updated, which will now need to be on the reflection call of the adapter. All the logic already exists for each of the modification types (deletes, adds, updates, mod DN) in the attribute changes method, which is currently used for deltas. Deltas do not actually use the adapter context, nor do they actually need to, as it is the value of the changes that are important, and these are not persisted in the adapter context. What we are looking at for this issue is actually "changes reflection", where the changes context should be updated in a similar way to the adapter context during reflection, as the original state of the context is not persisted. There are a few options for how this can be done:

• On reflection, persist the change types in memory - these changes will all be lost if the service is restarted, however, as the adapter context will be up to date for the current state of entities. A full import would be required for all adapters following a service restart in this case
• On reflection, persist the change types to the changes register via a hash, containing all the details of the change
• Persist the change types to the entity and entity value tables, with flags indicating an entity level add, update, mod DN, or delete, as well as flags indicating in the entity value table what the attribute level changes are

See IDB-151 for thoughts on the impacting race condition.

I think it would be useful to ensure that you layout what can be done under the following headings, to ensure that each criteria is met:

• Add
• Update
• Delete
• Modify DN
• Update attribute

The only option where the above would have different storage is under the third option above. For the changes register hash option, it would be stored directly to the change in the changes table. For the third option:

• adds would be flagged with a change type of add in the Entity table
• deletes would not be present in the adapter context, so changes present without a corresponding entry in the adapter context would be considered a delete
• updates would be flagged in the Entity table as having been updated, and each updated value would need to be marked as such in the Entity Value table
• Mod DN would be similar to update, where the DN would need to also be saved to Entity Value (I believe it may be already, but I would need to check again)

I think the second option of storing the changes in the changes register may be the faster option and reduce database traffic around this operation, plus it takes care of the persistence problems.

• Can you please outline how the second option would look?
• What change would be made to the database?
• How would this change accommodate the above four requirements.

Thanks.

After discussion with Adam, the following approach will be taken:

• Changes table updated with two additional columns
  • Change type - add, delete, update, modify DN --> an add will not need any other information, the current state of the entity will be written out
  • Previous DN - used for MOD DNs and deletes, otherwise the link to the existing entity cannot be made
• Entity Value table updated
  • A flag on each entity value to say whether or not its changed - a quick ADO.NET query at the end of the import to clear these flags will ensure the changes are cleared

According to RFC-2849, the removal of a single value from a multivalue field requires the previous value. Alternatively, the entire multivalue could be replaced by the "replace" statement in LDIF, eg.

The case of group management highlights that the second approach is actually much more efficient in the long run (ie. not having to spit out 199 groups if 1 from 200 is deleted). However, this case means that we need to preserve the old value of the field following the change. Alternatives:

• Have a change type field in the Entity Value table to retain whether or not an attribute has been added or removed - deletes would stick around until it was imported
• Have multiple rows in the changes table containing all attribute level modifications, eg. Remove - Telephone - 1234 5678, Add - Age - 32

The first option would allow us to minimize the row count in the changes table, where the second would remove the need to access the entity value table at all. My preference would be the first as we will already be accessing the value to see the entity, and we will not need additional rows in the changes table.

Update: The Entity Value table should contain an Add/Update/Delete flag on items, where by default items marked as delete are ignored by the main adapter context. Deltas would see that an entity is an "update" change type in the Changes table, access an adapter context where all change types were present, and spit out all th eadd, update and delete values to the LDIF file.

The database, and relevant upgrade scripts, will need to be updated when this behaviour changes.

Adam, I believe this is as complete as it can be for CTP. Assigning to you for appropriate action.

Note: The save behaviour should be looked at once we fix the pre-populate functionality.

Delta imports do not appear to be working currently. Things do not seem to be properly marked as pending changes according to the adapter statistics, but a full import into FIM will bring in the change.

Raising priority

The repository objects when processing connector pages for change detection, and the change processor job was not running on the schedule. Delta imports are now working again for Identity Broker, meaning the work for Alpha is now completed.

Reducing priority of remaining work, and moving to Beta.

Work done in IDB-1122.