Support for correct handling of rule vs. data changes in the FIM Portal

Bob Bradley 13 years ago updated by anonymous 9 years ago 7

I'm sure I've detailed this requirement in the former Project Management tool, but it is presently offline so I can't tell. I do recall spending quite some time writing this (and other issues) up in that tool - but I can't find them here. I feel strongly enough about this issue to write this up while on leave ...

A key requirement @ Centrelink (as with ANY FIM Portal implementation) is the ability for Event Broker to differentiate between data vs. rule changes that have occurred in the FIM portal, and thereby ensure that the correct operation list is run as a result. Both rule and data changes are imported together into the same FIM MA connector space (there can only ever be one FIM MA), and different sync cycles are required depending on whether or not the imported changes included rule changes (essentially sync rules).

The easiest way to explain this is in terms of a FIM administrator attempting to manually execute run profiles involving multiple MAs, including the FIM MA. If running a delta import/delta sync on the FIM MA immediately after a rule has changed in the portal, the FIM administrator should not be surprised to see the sync service raise a warning when he attempts to run a subsequent delta import/delta sync on another MA, with words to the effect that rules have been changed since the last full sync cycle, and that a new full sync is required. When the Event Broker (or any other non-interactive process) executes run profiles under this scenario a warning is written to the Application Event Log but this does not prevent the run profile from subsequently being executed.

A variation on this scenario has always been around since MIIS/ILM days, but what is new here is that the FIM portal now introduces an additional (less obvious) way that the sync engine configuration can be changed by a FIM administrator type. It is anticipated that there will be certain types of changes that could occur in day-to-day FIM operations whereby a FIM sync server under the "governance" of Event Broker would be expected to handle this scenario without having to first shut down the Event Broker service and manually perform "re-baselining".

I won't go into any further detail on this now, but suffice to say it was an issue for us @ Centrelink, and Jeff Nelson (MCS) sees this as a key business driver in the ultimate purchase of Event Broker by Centrelink.

The way change detection has been implemented for the FIM Portal @ Centrelink is using the file changes plug-in with log files written by a FIM custom activity (as I demonstrated in a live meeting on the NAB site before christmas) and the way that "data changes" are differentiated from "rule changes" are that the custom activity is used to write to either a "data" or a "rule" file depending on the nature of the change that has been logged. The challenge then is for Event Broker to correctly interpret changes to either file ...

The present Event Broker design does not allow multiple incoming operation lists to be configured for the same MA, but this would have allowed me to meet the requirement together with the mutex capability recently added to the product.

As a work-around I was able to configure the INCOMING operation list for the FIM MA using the standard File Changes plug-in for the DATA log file, and this did little more than a delta import/delta sync on the FIM MA (followed by a commit file changes plugin call). I coupled this with a GLOBAL operation list on a similar schedule to the above delta (10 secs which is the default) which ran the File Changes plug-in on the RULE log file as the first step in the run profile. If this plug-in returned false it "jumps to a label" operation at the end of the operation list, but proceeds to run a full sync on each MA (with each operation on the same MUTEX thread name as those of the regular delta operation list) if it returns true.

The implications now on the equal precedence issue (written up by Nigel this week) now throw up concern that running a series of full syncs to re-baseline the FIM sync service in ANY WAY could lead to loss of MV data for attributes marked as equal precedence applicable. However, this does not take away from the fact that data and rule changes detected in the FIM portal need to be handled differently.

Further to this, I am not really saying that we need to add support for multiple incoming operation lists for the same MA as part of release 3.0 ... rather that the story below needs to be catered for somehow, either by persisting or refining the workaround described below, or (as would be my preference) moving to an alternative of avoiding polling for log file changes altogether ... e.g. if instead of logging to a file in my custom activity I was instead able to invoke any nominated operation list in Event Broker directly (i.e. external invoking - another previous feature request that there has been talk of implementing soon) that wasn't necessarily the incoming operation list for the FIM MA, then that would suit me down to the ground!

Bob, you're right about it being less obvious in previous versions - you only really find out whether ILM/FIM wants a full sync by going in to the Event Viewer or attempting to run a manual delta sync operation, and of course, if significant change is made such that the service literally cannot continue without a full sync

It has been my impression that upon reconfiguring flow rules, attribute flows, or making a code change, a full sync should be run. At QDET, upon any of the above, a manual full sync is instigated as a part of the above change. This can be done with Event Broker running, as it will then continue to pick up changes and run deltas in the system once it has completed. Changes that are less significant are left to be dealt with by a weekly full synchronisation job on each management agent

As for FIM stating it requires a full sync job in the Event Viewer, I do not know if this is communicated via conventional means as a run profile status message. It would be interesting to see if this state is something that can be detected by Event Broker, but I would personally feel more comfortable with a manual full sync after a manual change is made.

I'll leave this for yourself and the rest of the product team to comment on in due course

Matt - 100% agreed on your first paragraph. The issue now (see the equal precedence story) is that when you have to do a full sync after making rule changes you get the unavoidable side-effect on (unwanted) equal precedence updates. There does in fact seem to be a way forward with this (see my latest comment on the equal precedence story). What you need to avoid @ DET (if you keep Event Broker running as you say) is premature export run profile execution before each MA has had a full sync done ... standard Event Broker config will cause this (unwanted behaviour) to happen by design. Hence the need for threading ...

On your 3rd paragraph there is DEFINITELY no communication back to the invoking process warning of the need for a full import/sync ... none at least that would enable you to abort the run (the event log is the only indication and that's a warning meaning the process continues regardless). What I am saying here is in that the ILM/MIIS world you should DEFINITELY manage the MANUAL resync after making config changes - but in the FIM Portal world we now have a distributed config admin model which means that it is far more likely that rule changes can occur for which the impact on the FIM sync server config was "out of sight, out of mind". I am not 100% certain either that such changes are limited to sync rule changes ... I have a feeling that the governing MPRs and Sets can trigger the same warning too, but I haven't got specific evidence of this at my disposal right now. You really need to get a feel for the new FIM Portal world to see what I mean here ...

I see what you mean about exports causing issues. It would be interesting to see if we can get a handle on the need for a full import/sync message somehow.

What would a response to this event be? An Event Broker invoked full sync? Or would it simply halt sync/export processing?

I know you're still on leave Bob, so don't worry if you can't get to this until you return - I'm answering now just in case you have some free time, as the earlier we have a potential solution the sooner we can get it done.

As I believe we've noted before, we're not interested in any functionality or features whatsoever that are the request of a single client and beneficial only to them or a very small portion of the market. In this instance you've already mentioned the story is applicable to anyone using the FIM portal which is fantastic, so we'll continue with the great evalatuion so far.

Having read everything a few times now I think finally understand the problems you're talking about and you've raised a few possibilities of solutions. As none of us have (nor can we gain in a reasonable timeframe) the knowledge of the FIM Portal we require to fully see this issue through, I recommend we have a short workshop upon your return next month to discuss some of our options. In the mean time, I'll post a few more thoughts as well as some of the changes we've incorported (or plan to include) in v3.0 so you can have a think about how they might affect (positively or otherwise) the solutions you've already outlined. I've listed them from what I perceive to be worst-to-best. Please correct me if I'm wrong on any of my assumptions.

Firstly, and I know you said this isn't the only requirement, but you can have as many operation lists of any type as you want in v3.0. Does this mean you could reconfigure your current approach of two files for data and rule changes respectively?

There's no such thing as Labels in v3.0. Whilst the technical implementation is still undergoing some changes, lists will essentially act more like trees, with operations being able to branch in to others based on particular decisions. The result (success/failure, etc.) of an operation is one such method of branching, and we're still looking in to whether other types of decisions are possible. Regarding your second point, it sounds as if having a single operation list that can break off in to two branches based on whether they are data or rule changes would work well. Is there a way we can determine this, either through SQL or WMI or some similar method? Looking at the Event Viewer programatically is not appropriate.

Finally, it sounds like you've said the best solution would be for an external source to be able to trigger operation lists in Event Broker. This is possible in v3.0, but will obviously have certain requirements in order for it to work. For example, we intend on exposing a WCF Endpoint that would accept these triggers. We spoke with Chris Cox the day after your presentation about whether or not it was possible to have some kind of FIM Portal custom task (excuse my poor terminology!) that we could implement that connects to our WCF endpoint and specifies which operation list should be executed based on some internal information, but unfortunately Adam and I quickly got lost in the FIM Portal talk, having never used it ourselves. Am I even close to the kind of "ideal solution" that you would be interested in?

Thanks again, get back to me when you can.

This story has long since been broken down in the components of EB-16. Set to closed.