Filtering data at the adapter level

Hi Bob,

A couple of questions to start with after talking to the others.

Firstly, when you talk about filtering at the connector level are you talking about connector-specific features like SQL connectors that allow modifying SQL statements? That's very different from a "connector filter" that applies to all kinds of connectors by importing everything in to Identity Broker and then excluding it from an adapter.

Secondly, we'd like to know more about why something like this would be useful to you. What advantages are there to filtering out entities at the adapter level? Why is a filter in FIM not more appropriate? I believe we've spoken before about the current company-defined purpose of Identity Broker and performing business logic such as this (transformations are more about modelling) is not something we want to include at this time.

Thanks for responding Patrick.

1. Yes - this is very much about the SQL style connector used @ DEEWR (and CSO).
2. Any FIM MA is a very poor place to apply a filter if it can be avoided, and must always be an absolute last resort where this could lead to a large number of (persisting) disconnectors. This is an ILM/FIM architectural consideration ... when you have a large number of disconnectors this causes the FIM Sync/ILM service to re-attempt to join them to the metaverse (thereby making them a connector) on EVERY SUBSEQUENT SYNC CYCLE (not only full, but delta as well). In the DEEWR case a filter WAS being used at the FIM MA level until we realized that the performance overhead was a show stopper, and resorted to a connector filter instead (after confirming the filtered records served no purpose in the sync design).

I spoke about this with everyone shortly after your comment and just forgot to update the issue.

We'll definitely be open to having a more thorough discussion about it, but we decided there's too many difficulties to even consider doing this in the current project, so we'll review it when 4.1 rolls around.

This issue might have some relation to IDB-950, depending on how it's done. A discussion must still be had as to whether this is even desirable, however.

Hi Bob Bradley, Matthew Clark has suggested that FIM R2 declared connector filter (possibly the wrong name) may solve this issue for you. Have you come across it?

Tony Sheehy - I understand what Matthew Clark is talking about - yes this is a filter applied to a declarative inbound sync rule which allows you to limit the scope of your MA (and yes, hence your adapter). However there are 2 problems:

1. This only applies to declarative rules, and these seem somewhat out of favour right now in most sectors;
2. Even with this filtering you still have to populate a connector space unnecessarily (including for ECMA1 building a larger-than-necessary LDIF file, and including unwanted overhead in network traffic); and (more importantly)
3. This is likely to incur a performance overhead at sync time, similar to standard MA filters which process disconnectors unnecessarily.

So no, Tony - nice thought, but this is definitely not what I am after, and I still see the need for this concept, particularly with MAs that only need a small number of the base connector attributes..

Hey Bob, the thing I was talking about is the Declared (Import Filter) new in R2, it actually prevents things from entering the connector space completely even in a non-declarative environment. It means that all the syncs run without filtered disconnectors too - see [

Ah ... well if that's the way it works, Matthew Clark then that's awesome. I had heard about the feature but (not using R2 anywhere currently) wasn't aware of the key observation you just did when you said "prevents from entering the connector space". Makes sense - a feature we've only ever had in the AD MA before with OU filtering.

This being the case, then for IdB 4.* with FIM R2 that would make my statements #1 and #2 incorrect , leaving #3 the only reason to implement this feature. Of course with ILM and FIM R1 all 3 statements hold still - and we will see FIM R1 sites for some time to come yet (it's on a par with the significance of moving from IdB 3.* to 4.*) I can still see value in this feature.

From an "eliminating noise and overhead" perspective, cluttering up the LDIF file (ECMA#1) will still be valuable, and even moving to ECMA2 when we still are likely to want to support the file-based import option.

Confirmed or migrated to VSO.