Identity Broker Forum
Welcome to the community forum for Identity Broker.
Browse the knowledge base, ask questions directly to the product group, or leverage the community to get answers. Leave ideas for new features and vote for the features or bug fixes you want most.
Intelligent cycles for polling and non-polling connectors
With
- any Identity Broker deployment
- polling or non-polling
- with or without Event Broker, and
- for whatever version
as an implementor you are always making little more than an educated guess as to the appropriate cycle of full and/or delta imports for each of your connectors. This needs to be more scientific, and an opportunity may exist as part of Identity Broker 4 to take empirocal data and suggest refinements (thinking green/yellow/red dashboard style info here) on what would make optimal use of available CPU/network resources.
Equally, with frequencies recently configured for CSODBB's Peoplesoft (polling) connector for PHRIS, we found that my initial values were on the over-ambitious side. Something to draw attention to the fact that the service was "spinning its wheels" trying to keep up with unrealistic cycles would be useful console feedback (i.e. I summised that the number of queued but unprocessed polling requests was growing because they couldn't be processed fast enough). Ryan had some trouble and called me about it during UAT last week, where memory for the Identity Broker service grew astronomically and delta imports started failing. In the end I think that the resolution was at least partly to do with setting realistic frequencies.
Redundant image node for AdapterConfiguration in CompositeAdapterConfiguration
The image node for a composite adapter renders the image node for any contained adapters redundant. However, when the adapter node is excluded from the adapter xml the Identity Broker service fails to start - and an error message is displayed stating that the node is mandatory. When an empty node is added the service still fails to start, but another exception is raised instead. The service will only start when a non-empty image node is included in the configuration.
I noticed the presence of these images (which are not displayed in the Identity Broker Management Studio) when building xsl stylesheets to document the Identity Broker configuration for DEEWR. Not only do they add unnecessary bulk to the configuration, but they can lead to irrelevant images persisting and being accidentally deployed (the images that I found were actually carried over from another project).
As a work-around I have generated a dummy binary string from the smallest PNG file I could find, and used that for all of the adapters that make up my composite adapter.
I believe the image node should only be mandatory for an adapter if it is NOT part of a composite adapter. Given that composite adapters are now likely to be the norm rather than exception, certainly when used with FIM, then this issue is likely to affect more deployments. I doubt whether anyone who has deployed a composite adapter actually realizes what images lay hidden in the nested adapter configurations they have deployed.
Ideas for improved Identity Broker configuration exception reporting
I've noticed that in general the exception reporting is very good at identifying the cause of a problem, however in the following scenarios it is not:
- when 2 field nodes in the same entitySchema have the same name (obvious error but easy to make when hand crafting xml) the exception raised in the Application Event log (when the Identity Broker fails to start) is simply "The parameter is incorrect". There is no evidence as to what was the problem, nor whether it was an adapter or connector issue;
- when configuring a Relation.Group.Composite transformation, I accidentally included a key reference to a column of the base connector in a dnComponent, instead of a column defined by the RelationshipConnectorID - in this case the IdB service started OK, but when attempting an Adapter entity search an exception "Adapter get all entities for adapter xxx failed with reason 'Specified argument was out of the range of valid values. Parameter name: attributeValue'". Of course there is no such parameter "attributeValue" exposed in the adapter config, so I presume this is internal to IdB. While the text makes sense once you know the problem is with your DN, trying to track this problem down in a composite adapter with many adapters configured is quite problematic.
I'm sure if I was diligent in logging more of these in JIRA I would come up with a few more, so maybe we can keep reusing this JIRA item in the future ... but right now the above 2 are a good start
Potential issue with changes register being cleared regardless of delta import success
At QDET in a mirror production environment we recently saw an issue around the changes register of Identity Broker. A full import was running on a large connector (500000~ users) where 40 changes in the target system were present. The Identity Broker Changes Plug-in detected a change during the import process and kicked off two delta imports into FIM. Possibly due to poor infrastructure or heavy database server load, the delta imports failed - logs below:
20111013,05:28:02,Adapter request to get attribute changes from adapter space.,Adapter,Information,Adapter request to get attribute changes from adapter space 53e85508-7648-409c-b451-0769028bba70.,Normal
20111013,05:28:05,Started processing changes register items.,Change detection engine,Information,Started processing changes register items for connector IRegister Person.,Normal
20111013,05:28:05,Changes register item processing completed.,Change detection engine,Information,Changes register item processing on connector IRegister Person completed. Duration: 00:00:00.0937500,Normal
20111013,05:28:41,Adapter request to get entity from adapter space failed.,Adapter,Warning,"Adapter request to get attribute changes from adapter space 53e85508-7648-409c-b451-0769028bba70 failed with reason The transaction is in doubt.. Duration: 00:00:39.3281250
Error details:
System.Transactions.TransactionInDoubtException: The transaction is in doubt. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadBuffer()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at System.Data.SqlClient.SqlDelegatedTransaction.SinglePhaseCommit(SinglePhaseEnlistment enlistment)
— End of inner exception stack trace —
at System.Transactions.TransactionStatePromotedIndoubt.PromotedTransactionOutcome(InternalTransaction tx)
at System.Transactions.CommittableTransaction.Commit()
at System.Transactions.TransactionScope.InternalDispose()
at System.Transactions.TransactionScope.Dispose()
at Unify.Framework.UnifyTransactionScope.Dispose()
at Unify.Data.LinqContextConversionBase`4.SubmitChanges()
at Unify.Repository.AdapterEntityPartitionUpdatableContextAdapter.SubmitChanges()
at Unify.Framework.Adapter.ProcessAttributeChangePage(IEnumerable`1 pageOfChangedIds)
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at Unify.Framework.EnumerableExtensions.<ActionOnLast>d__16`1.MoveNext()
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at Unify.Framework.EnumerableExtensions.<ActionOnFirst>d__1c`1.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Unify.Framework.ActionOnExceptionEnumerator`1.MoveNext()
at Unify.Framework.EnumerableExtensions.<ActionOnLast>d__16`1.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at System.Linq.Enumerable.<ConcatIterator>d__71`1.MoveNext()
at Unify.Framework.LDIFComponentFileGenerator`1.GenerateFile(TextWriter writer, IEnumerable`1 entries)
at Unify.Framework.LDIFAdapter.<>c_DisplayClass5`1.<CreateLDIFComponentStream>b_4(Stream stream)
at Unify.Framework.LazyEvaluationStream.Evaluate(Object obj)",Normal
20111013,05:28:50,Connector processing success.,Connector Processor,Information,"Processing page 8 for connector IRegister Person processed 1875 entities, finding 5 differences. Duration: 00:00:48.6562500",Normal
20111013,05:28:50,Connector Processing started.,Connector Processor,Information,Connector Processing started for connector IRegister Person (page 9),Normal
20111013,05:28:50,Started processing changes register items.,Change detection engine,Information,Started processing changes register items for connector IRegister Person.,Normal
20111013,05:28:52,Changes register item processing completed.,Change detection engine,Information,Changes register item processing on connector IRegister Person completed. Duration: 00:00:01.5000000,Normal
20111013,05:28:54,Adapter request to get attribute changes from adapter space.,Adapter,Information,Adapter request to get attribute changes from adapter space 53e85508-7648-409c-b451-0769028bba70.,Normal
20111013,05:28:56,Connector processing success.,Connector Processor,Information,"Processing page 9 for connector IRegister Person processed 2000 entities, finding 0 differences. Duration: 00:00:06.1718750",Normal
20111013,05:28:56,Connector Processing started.,Connector Processor,Information,Connector Processing started for connector IRegister Person (page 10),Normal
20111013,05:28:56,Get all entities from connector completed.,Connector,Information,Get all entities from connector IRegister Person return 18412 entities. Duration: 00:02:01.6875000,Normal
20111013,05:29:10,Connector processing success.,Connector Processor,Information,"Processing page 10 for connector IRegister Person processed 1900 entities, finding 0 differences. Duration: 00:00:13.6093750",Normal
20111013,05:29:10,Connector Processing started.,Connector Processor,Information,Connector Processing started for connector IRegister Person (page 11),Normal
20111013,05:29:14,Connector processing success.,Connector Processor,Information,"Processing page 11 for connector IRegister Person processed 1012 entities, finding 0 differences. Duration: 00:00:04.3281250",Normal
20111013,05:29:32,Adapter request to get entity from adapter space failed.,Adapter,Warning,"Adapter request to get attribute changes from adapter space 53e85508-7648-409c-b451-0769028bba70 failed with reason The transaction is in doubt.. Duration: 00:00:38.0937500
Error details:
System.Transactions.TransactionInDoubtException: The transaction is in doubt. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadBuffer()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at System.Data.SqlClient.SqlDelegatedTransaction.SinglePhaseCommit(SinglePhaseEnlistment enlistment)
— End of inner exception stack trace —
at System.Transactions.TransactionStatePromotedIndoubt.PromotedTransactionOutcome(InternalTransaction tx)
at System.Transactions.CommittableTransaction.Commit()
at System.Transactions.TransactionScope.InternalDispose()
at System.Transactions.TransactionScope.Dispose()
at Unify.Framework.UnifyTransactionScope.Dispose()
at Unify.Data.LinqContextConversionBase`4.SubmitChanges()
at Unify.Repository.AdapterEntityPartitionUpdatableContextAdapter.SubmitChanges()
at Unify.Framework.Adapter.ProcessAttributeChangePage(IEnumerable`1 pageOfChangedIds)
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at Unify.Framework.EnumerableExtensions.<ActionOnLast>d__16`1.MoveNext()
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at Unify.Framework.EnumerableExtensions.<ActionOnFirst>d__1c`1.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Unify.Framework.ActionOnExceptionEnumerator`1.MoveNext()
at Unify.Framework.EnumerableExtensions.<ActionOnLast>d__16`1.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at System.Linq.Enumerable.<ConcatIterator>d__71`1.MoveNext()
at Unify.Framework.LDIFComponentFileGenerator`1.GenerateFile(TextWriter writer, IEnumerable`1 entries)
at Unify.Framework.LDIFAdapter.<>c_DisplayClass5`1.<CreateLDIFComponentStream>b_4(Stream stream)
at Unify.Framework.LazyEvaluationStream.Evaluate(Object obj)",Normal
20111013,05:29:35,Change detection engine import all items completed.,Change detection engine,Information,Change detection engine import all items for connector IRegister Person completed. Duration: 00:02:44.3593750,Normal
20111013,05:29:47,Adapter request to get attribute changes from adapter space.,Adapter,Information,Adapter request to get attribute changes from adapter space 53e85508-7648-409c-b451-0769028bba70.,Normal
Subsequent delta imports into FIM were successful, however, the import returned 0 results. This may suggest that the changes register is cleared regardless of the return state of a delta import. A full import into FIM was required to pick up the changed users.
Attempts to replicate this behaviour have so far been unsuccessful - the database is no longer timing out.
This is not a pressing issue as we have been unable to replicate yet (and may not be able to), but it would be worth investigating (for this or future versions) to prevent this from occurring again as the behaviour does incur operational intervention.
More debug information for Identity Broker
In order to aid diagnosis of failing processes, I think it would be a good idea for there to be a configurable option to provide detailed diagnosis information at every interface boundary within Identity Broker.
By that, I mean every Identity Broker process that has interfaces should be able to have a decorator inserted to provide details information about the methods and data provided at each step of a process. This would prevent the kinds of Product Support issues where: "I can't really tell what's going on so I suspect something is wrong with Identity Broker".
Multivalue DN generator transformation contribution breaks
When the relational connector is updated for the multivalue DN generator transformation, the changes register errors with the following:
Changes register item processing on failed.
Changes register item processing on connector CSV Test failed with reason The column GroupMulti cannot be used to form a chained transformation.. Duration: 00:00:00.0322266
Error details:
Unify.Product.IdentityBroker.ColumnBlacklistedException: The column GroupMulti cannot be used to form a chained transformation.
at Unify.Product.IdentityBroker.MultiValueSourceEntityDistinguishedNameGeneratorTransformationFactory.ApplyChangeDetectionColumnInformation(IAdapterColumnSources columnInformation)
at Unify.Framework.Visitor.VisitT(IEnumerable`1 visitCollection, Action`2 visitor)
at Unify.Product.IdentityBroker.AdapterEngine.CreateColumnSources(IAdapterEntityTransformationFactory factory, IEntitySchema baseSchema, Guid baseConnectorId, String adapterName, Guid adapterId)
at Unify.Product.IdentityBroker.AdapterEngine.<>c_DisplayClass27.<GenerateAdapter>b_23()
at Unify.Product.IdentityBroker.ChainedTransformationChangeProcessor.PublishChange(IEnumerable`1 changedEntities, DateTime changeProcessTime, ICollection`1 changeRecords)
at Unify.Product.IdentityBroker.ChainedTransformationChangeProcessor.ProcessChangeReport(IDictionaryTwoPassDifferenceReport`4 changesReport, DateTime changeProcessTime)
at Unify.Framework.Visitor.VisitT(IEnumerable`1 visitCollection, Action`2 visitor)
at Unify.Product.IdentityBroker.ChangeReportProcessor.CreateAndProcessReportT(IEnumerable`1 adapterTransformationProcessors, IEnumerable`1 sourceEnumerable, DateTime changeTime, Action`2 addAction)
at Unify.Product.IdentityBroker.ChangeReportProcessor.ProcessReport(IChangeReportProcessingRequest request)
Cannot promote an adaptor to a composite adaptor
When promote SAPPerson adaptor to SAP composite adaptor, an error is displayed to the browser:
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at ASP._Page_Views_Adapter_Transformations_Transformations_cshtml.Execute() in c:\Program Files\UNIFY Solutions\Identity Broker\Web\Views\Adapter\Transformations\Transformations.cshtml:line 90
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
at System.Web.Mvc.RazorView.RenderView(ViewContext viewContext, TextWriter writer, Object instance)
at System.Web.Mvc.BuildManagerCompiledView.Render(ViewContext viewContext, TextWriter writer)
at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResult(ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.<>c_DisplayClass1c.<InvokeActionResultWithFilters>b_19()
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func`1 continuation)
at System.Web.Mvc.ControllerActionInvoker.<>c_DisplayClass1c.<>cDisplayClass1e.<InvokeActionResultWithFilters>b_1b()
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)
CompositeAdaptorIssue.JPG
Page 1 is missing from logs
Change detection and other logs using paging seem to start from page 2 - there is no page 1 in the log. See below for an example:
29/May/2012 09:22:30 Information
Connector Import all entities from connector completed.
Import all entities from connector CSV Test.txt return 4 entities. Duration: 00:00:00.0380860
29/May/2012 09:22:30 Information
Connector Processor Connector Processing started.
Connector Processing started for connector CSV Test.txt (page 2)
29/May/2012 09:22:31 Information
Connector Processor Connector processing success.
0 entites in cumulative total. Current processing of page 2 for connector CSV Test.txt processed 4 entities, finding 2 differences. Duration: 00:00:01.5244140.
29/May/2012 09:22:31 Information
Change detection engine Change detection engine import all items completed.
Change detection engine import all items for connector CSV Test.txt completed. Duration: 00:00:02.1455078
Last Import always denoted as failure on home page if errored
The last errored state of a connector is not cleared on start of that job. This means that if that job has errored before, when inactive the background-colour will be red.
Customer support service by UserEcho
