0
Completed

Request for deletion threshold to prevent mass deletion on imports

Richard Green 5 years ago • updated by anonymous 4 years ago 5

Hey Guys,

This request has come as a result of the deletion issues experienced at APRA on the weekend.

Is it possible to implement a feature to abort a connector import / change detection run if an unexpected number of deletes are imported?

I'm thinking something along the lines of a flag to enable/disable, and a threshold value (either a number of entities, or percentage value).
Ideally this would be configurable per connector (as some may legitimately and regularly experience large changes in existing data).

This kind of issue has happened previously on other sites, and a feature like this could prevent temporary data issues within a source system from having impact on downstream systems.

Affected Versions:
Fixed by Version:

Update threshold too please!

There is an additional case when OptimalIDM's VDS dropped one of the 4 downstream LDAP systems at DSITIA causing similar issues.

For Origin I just had every user object in the adapter temporarily lose their address details due to a broken link (underlying key values changed in source connector data). Due to unavoidable connector full import timing delays, referential integrity was not re-established in time to prevent deletion of user attribute values in FIM, triggering unwanted workflow policy.

I also had this happen in CSODBB's production FIM solution (IdB 3.* for PeopleSoft PHRIS connector) - see critical incident CSODBB-334. While a mitigation strategy was put in place to ensure these types of source system bulk data changes occurred at times Identity Broker was not running, this remained a significant risk. This turned out to be the case when an HR batch process was initiated in business hours (rather than the specified outage window) causing the same result as before - CSODBB-411.

Adam van Vliet - I think weight of incidents is now at the point where the need for such a feature has become critical.

Richard Green - thinking of the HPSM connector for Origin, and the relatively high likelihood that either the wrong file or an empty file will be dropped in the target folder, what is the best way we can provide this sort of safety mechanism in the initial Origin release (IdB 4.1)? I am thinking that we could use Event Broker to separate out the DI from the DS and run a CSEXPORT in between to check for change volumes against a threshold? This could actually give rise to a generic FIM approach before we have the feature in IdB ...