0
Answered

Filtering data at the adapter level

Bob Bradley 7 years ago in UNIFYBroker/Microsoft Identity Manager • updated by anonymous 3 years ago 9

While it is often useful to be able to filter records at the connector level, it would be handy to be able to do this at the adapter level as well.

An example is one which occurred at DEEWR where I needed to exclude all CLAIM objects from the Claims adapter where the IsDerived flag was set to 1. Since this was a single adapter/single connector configuration, this was easily achieved at the connector level. However, the side-effect was that this same connector was also being used in another adapter with a different base connector ... to derive group membership style reference properties for a PERSON object. There was one such transformation that needed ALL claims objects (i.e. inclusive of the IsDerived=1 claims) in order to calculate the membership. In my case I had already decidedd to discontinue using the group transformation and achieve the requirement a different way, but this could have forced me down the path of multiple connectors (10s of 1000s of rows) for the same data source.

If this is not already achievable (without configuring multiple connectors for the same data source), please consider this as a feature request.


Declared Import Filter.png
Affected Versions:
Fixed by Version:

Hi Bob,

A couple of questions to start with after talking to the others.

Firstly, when you talk about filtering at the connector level are you talking about connector-specific features like SQL connectors that allow modifying SQL statements? That's very different from a "connector filter" that applies to all kinds of connectors by importing everything in to Identity Broker and then excluding it from an adapter.

Secondly, we'd like to know more about why something like this would be useful to you. What advantages are there to filtering out entities at the adapter level? Why is a filter in FIM not more appropriate? I believe we've spoken before about the current company-defined purpose of Identity Broker and performing business logic such as this (transformations are more about modelling) is not something we want to include at this time.

Thanks for responding Patrick.

  1. Yes - this is very much about the SQL style connector used @ DEEWR (and CSO).
  2. Any FIM MA is a very poor place to apply a filter if it can be avoided, and must always be an absolute last resort where this could lead to a large number of (persisting) disconnectors. This is an ILM/FIM architectural consideration ... when you have a large number of disconnectors this causes the FIM Sync/ILM service to re-attempt to join them to the metaverse (thereby making them a connector) on EVERY SUBSEQUENT SYNC CYCLE (not only full, but delta as well). In the DEEWR case a filter WAS being used at the FIM MA level until we realized that the performance overhead was a show stopper, and resorted to a connector filter instead (after confirming the filtered records served no purpose in the sync design).

I spoke about this with everyone shortly after your comment and just forgot to update the issue.

We'll definitely be open to having a more thorough discussion about it, but we decided there's too many difficulties to even consider doing this in the current project, so we'll review it when 4.1 rolls around.

This issue might have some relation to IDB-950, depending on how it's done. A discussion must still be had as to whether this is even desirable, however.

Hi Bob Bradley, Matthew Clark has suggested that FIM R2 declared connector filter (possibly the wrong name) may solve this issue for you. Have you come across it?

Tony Sheehy - I understand what Matthew Clark is talking about - yes this is a filter applied to a declarative inbound sync rule which allows you to limit the scope of your MA (and yes, hence your adapter). However there are 2 problems:

  1. This only applies to declarative rules, and these seem somewhat out of favour right now in most sectors;
  2. Even with this filtering you still have to populate a connector space unnecessarily (including for ECMA1 building a larger-than-necessary LDIF file, and including unwanted overhead in network traffic); and (more importantly)
  3. This is likely to incur a performance overhead at sync time, similar to standard MA filters which process disconnectors unnecessarily.

So no, Tony - nice thought, but this is definitely not what I am after, and I still see the need for this concept, particularly with MAs that only need a small number of the base connector attributes..

Hey Bob, the thing I was talking about is the Declared (Import Filter) new in R2, it actually prevents things from entering the connector space completely even in a non-declarative environment. It means that all the syncs run without filtered disconnectors too - see [

Ah ... well if that's the way it works, Matthew Clark then that's awesome. I had heard about the feature but (not using R2 anywhere currently) wasn't aware of the key observation you just did when you said "prevents from entering the connector space". Makes sense - a feature we've only ever had in the AD MA before with OU filtering.

This being the case, then for IdB 4.* with FIM R2 that would make my statements #1 and #2 incorrect , leaving #3 the only reason to implement this feature. Of course with ILM and FIM R1 all 3 statements hold still - and we will see FIM R1 sites for some time to come yet (it's on a par with the significance of moving from IdB 3.* to 4.*) I can still see value in this feature.

From an "eliminating noise and overhead" perspective, cluttering up the LDIF file (ECMA#1) will still be valuable, and even moving to ECMA2 when we still are likely to want to support the file-based import option.