We recently encountered an issue where a significantly large connector import contained a user with some bad data which mismatched its expected schema type (Identity Broker schema expected an integer, where the target system fed in the value "1.00E+01"). This is obviously quite frustrating, especially if this could happen at the end of a large import from a critical system, where large amounts of changes are expected. This is the behaviour of most of our current connectors (if not all, eg. Chris21, SharePoint).

An alternative considered was to individually log the entities which contain bad data, and continue the import. However, this has further reaching implications:

• If the Identity Broker logs are not routinely monitored, entities containing bad data could be completely ignored in the solution
• If a user is provisioned through the identity management solution, and at a later stage has bad data introduced, the user could be removed by a connector full import, and would then flow through as a delete through the solution

Identity Broker currently does not have a "strong" mechanism for informing users of events such as these outside of its current behaviour of halting the entire import. An alerting system similar to Event Broker 3 could prove useful, but still faces the dilemma of being ignored if not routinely monitored.

Can we make any improvements to address this issue?

A small issue I noticed with IdB is that when logging to the Windows event log and logging is set for severity of Information, the text in the event is not correct for the event type, the event text for non-error events is prefix with "an error occurred in module" as below. Small thing.

Log Name: Application
Source: Changes register item processing completed.
Date: 11/04/2011 11:59:22 AM
Event ID: 0
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: EDTIDM001VS.test.act.gov.au
Description:
*Error occurred in module: *Change detection engine

Changes register item processing on connector MAZE HUGHES PRIMARY Class connector completed. Duration: 00:00:00.3906200

Basically - What's the best way to blow away an IdB database and start with a fresh one? Can I do it without having to re-install all the add-ons?

Backstory - During testing I switched out the adapters and connectors for ones that connected to csv files instead of the end point systems and switching them back has scrambled everything. PartitionsIDs in IdB or something. After running the clear SQL database script in the IDB install directory the service wouldn't start - said bad db connection - but strangley the service debugger ran fine. I found another copy of the IDB database in sql named differently so I added the connection string for that one and the service started working. I used the clear sql script on that one and now i'm running a full import since about 11 this morning - so about five hours. I'm pretty sure there's about 20000 users but I don't think it would take that long just to get into the connector. My question is above in the basically part.

A validator "short cut" is required so that in connector configuration files a column can be given the type of short. This was required when importing from a SQL Server DB with a smallint value.

Shane attempted to create one but there were problems with this deeper in the framework to do with IComparable methods not being implemented for a particular data type or something. Below is the code he tried and the errors returned.

public class ShortValueValidatorFactory : Unify.Framework.ValidatorFactory.RangeValidatorFactoryBase<ShortValue, int>
{

protected override ShortValue GetValue(System.Xml.Linq.XName attribute, ShortValue missingValue, System.Xml.Linq.XElement fieldXml)

protected override ShortValue MissingMaxValue()

protected override ShortValue MissingMinValue()

}
}

Error 1 The type 'Unify.Framework.ShortValue' cannot be used as type parameter 'TValue' in the generic type or method 'Unify.Framework.ValidatorFactory.RangeValidatorFactoryBase<TValue,T>'. There is no boxing conversion from 'Unify.Framework.ShortValue' to 'System.IComparable<Unify.Framework.ShortValue>'. \\VBOXSVR\VMShare\DETSAFEPersonConnector\DETSAFEPersonConnector\DETSAFEPersonConnector\Connector\Unify.Connector.DETSAFEPerson\ShortValueValidatorFactory.cs 9 18 Unify.Connector.DETSAFEPerson

Error 2 The type 'Unify.Framework.ShortValue' cannot be used as type parameter 'TValue' in the generic type or method 'Unify.Framework.ValidatorFactory.RangeValidatorFactoryBase<TValue,T>'. There is no boxing conversion from 'Unify.Framework.ShortValue' to 'Unify.Framework.IValue<int>'. \\VBOXSVR\VMShare\DETSAFEPersonConnector\DETSAFEPersonConnector\DETSAFEPersonConnector\Connector\Unify.Connector.DETSAFEPerson\ShortValueValidatorFactory.cs 9 18 Unify.Connector.DETSAFEPerson

Currently, when importing on a connector, Identity Broker will generate a duplicate key error when it encounters multiple records that have the same key. This is problematic in itself, but is exacerbated because IdB apparently drops the whole page (~1000 records) when this occurs.

The error presented is something like this:
System.ArgumentException: An item with the same key has already been added.

Given that the impact of having a single duplicate record is quite high (1000 records not imported!), it's important to find these duplicate records in the end source and to either eradicate them, or select a more appropriate key value. To this end, I request an improvement for IdB to display the duplicate key in the error message when the error is thrown. Eg,

System.ArgumentException: An item with the same key (12345) has already been added.

Or perhaps even go so far as to include the attribute(s) used in the key:

System.ArgumentException: An item with the same key (entityID:12345) has already been added.
or (in the case of multiple attribute keys
System.ArgumentException: An item with the same key - (entityID,12345):(locationID,123) - has already been added.

I have not played with IdB nearly enough a, nor read the manual enough times, so please be gentle with me if any of these features already exist.

"Search" entities in connector or adapter - it would be great to have an option to enter search criteria first rather than have to list all then sort.
It is probably not so bad in many sites, but DET as an example, takes an age to load 109,000 entities, which is a pain when we only want to look at one.

It would also be really nice to have an option to do a limited import based on similar criteria (so as well as full import or delta import to have an option to import all sn=smith or something. That is of most use during set up and troubleshooting so maybe an "admin" feature.

I guess what would achieve the same (and be more operationally useful) would be a filter option to include or exclude specific entities - based on attributes and built up like an SQL query or an LDAP query.

Configuration via a web front end rather than by editing XML files - yipee!

Scheduling.
The polling intervals being set in ticks is horrible, so something in seconds or minutes would be far nicer.
Better still would be some form of schedulting in the product that offers more than get full import every 30 minutes, get deltas every 5 mins etc.
Something along the lines of the scheduling that can be set up in EB would be great, but even if there was just the option to not run on certain days or between certain times.
I guess a hook into EB would work too.

GUIDs
I was talking to Nick Mathas about a problem in the Novell world, where clearing the connector also clears the GUIDs which stuffs up the Novell IDM association value (unique key). He and I think this is something that needs to be addressed in the NIM adapter or the NIM end of the system rather than the IdB engine, but would there be a possibility to have a configuration item that could allow the source system unique key to be used as the IdB unique key in place of the IdB GUID?
i.e. If you do not select a unique key, you get IdB generated GUID and clearing the Connector and re-importing will generate new keys, but if you have selected that, for example, detnumber from a Chris connector should be the unique key, it will use that (relying on the source to guarantee uniqueness) and clearing the connector and re-importing will bring entities in exactly the same.

That's all the springs to mind for now

Optionally allow Relational.Compare.String to exclude items not in priority list. This was raised as an issue on BCE-220.

In order to prevent lots of scrolling on the adapter schema pages, allow collapsing of non-key fields.

An alternative to having to change batch size/timeout settings on export, it might be worth considering adopting a similar approach that Microsoft did with the MIM (Service) MA - i.e. changing the default option to asynchronous instead of synchronous exports.

Under the current default configuration, the MIM MA gets a success returned from the MIM Service once an exported change has been successfully queued (inserted in the REQUEST - either single or batch request objects). A similar approach might be worth considering for IdB such that we can decouple long-running connector export times from the MIM export itself.

I am categorising this request under O365 because that is where I am seeing the most need for this feature right now - however this would be a generic option.