Export Performance

Rizwan Ahmed 10 years ago in UNIFYBroker/Google Apps updated by anonymous 8 years ago 2

We did set a bench mark for 10,000 exports/hour – which translates to about 8 exports per second.
The current speeds for exports are at 5-7 users per minute. We would expect out peak student intake to take ~24 hours to process.
I did call out very early in the piece that performance was important, but tricky to get right. No consulting has been done with Monash on the export process or results shared of IdB internal benchmarks. As it stands, this is a show stopping issue.

The comparison is not valid. A single export is made up of more than just a single call to Google. For example, a user update: Identity Broker retrieves the item being updated from its internal store; if a flattened field is being updated, the user is retrieved from Google (patch semantics are not possible if this is the case); the user is then patched (or updated if required, which is slower); aliases are then added or removed as required; the user is made admin if required.
In addition to that, the logging in IdB can be set to Verbose, which will log the start and end of the requests to Google

In the next major version of Identity Broker we plan on moving to an ECMA2 based MA, allowing us to optimise the retrieval from the Identity Broker internal store.

RA 10/10: UNIFY is working on it, there are a few places where we could trade solution integrity for speed, such as fire and forget for the secondary calls (e.g. wait for the response from add user method, then fire and forget the add aliases method).

Resolved/closed. Accepted by Monash.