Identity Broker Forum
Welcome to the community forum for Identity Broker.
Browse the knowledge base, ask questions directly to the product group, or leverage the community to get answers. Leave ideas for new features and vote for the features or bug fixes you want most.
Return error to MIM when exporting via a PowerShell connector
Is there a way to return an error to MIM for individual entities while exporting via a PowerShell connector? My export script uses the standard try/catch and pushes failures where an entity fails to export, eg:
try{
#Export entity
}catch{
$components.Failures.Push($entity);
}
Though when an entity fails it returns a very generic error to MIM.
Is there a way to return a more descriptive along with each of the individual failures?
Register Contributions where the contributing attribute is from a joining connector?
Is it possible to call register-contribution in an adaptor PowerShell schema, where the source field is not from the base connector, but a joining connector?
I have been working on fixing the Register-contribution functions in a customer environment, however some of the values that are imported and are eventually used for time offset flag calculations, are coming from a non-base connector using the "Join on" transformation.
When I test using an imported change from the base connector, the changes are schedule for the correct time. When I test using an imported change a joining connector, my test fails (the imported change flows to the adaptor but does not appear to register the future change). This leads me to the conclusion that maybe an imported change from joining connector does not register future changes.
Is this correct? Is there a solution given termination/end dates required for some of the calculations, are not currently available on the base connector?
This has been implemented and is available in the release of UNIFYConnect V6, which will be made available shortly.
A patch for 5.3 is available on request.
PowerShell group connector returning null for dn attribute
Version v5.3.2 Revision #0
I have a PowerShell connector that queries and database to build groups including their memberships. But when importing it returns the following error. The connector does not have an associated adapter.
I have another connector that uses the same script that works just fine, which would indicate a data issue however, there are no obvious fields that are null.
Connector Processor Connector processing failed.
Connector Processing page 1 for connector Test Group Errors failed with reason Value cannot be null.
Parameter name: dn. Duration: 00:00:18.4144781.
Error details:
System.ArgumentNullException: Value cannot be null.
Parameter name: dn
at Unify.Framework.IO.DistinguishedName.op_Implicit(DistinguishedName dn)
at Unify.Product.IdentityBroker.Repository.EntityDistinguishedNameValueDataUtility`1.ConvertValueToString(DistinguishedNameValue value)
at Unify.Product.IdentityBroker.Repository.StringBasedValueDataUtilityBase`2.SetEntityValue(__EntityValueInsertRow dataValue, TValue value)
at Unify.Product.IdentityBroker.Repository.EntitySingleValueDataUtilityBase`2.CreateEntityValue(TEntityKey key, IValue value, IEntityCollectionKeyUtility`1 collectionKeyUtility, EntityDataSet set, __EntityInsertRow row, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.ConvertEntityValueToDataValue(KeyValuePair`2 entityValueAndKey, __EntityInsertRow row, EntityDataSet entityDataSet, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.<>c__DisplayClass33_0.b__0(KeyValuePair`2 entityValueAndKey)
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.d__17`2.MoveNext()
at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.InsertItems(ISet`1 addedItems, EntityDataContext sourceContext, SqlConnection connection)
at Unify.Framework.Data.LinqContextConversionBase`4.SubmitChanges()
at Unify.Product.IdentityBroker.SaveChangedEntitiesTransformationUnit.Transform(IDictionaryTwoPassDifferenceReport`4 input)
at Unify.Product.IdentityBroker.ConnectorEntityChangeProcessor.ProcessEntities(IEnumerable`1 connectorEntities, IEnumerable`1 repositoryEntities, IEntityChangesReportGenerator`2 reportGenerator)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_1.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()
Change detection engine Change detection engine import all items failed.
Change detection engine import all items for connector Test Group Errors failed with reason An error occurred while evaluating a task on a worker thread. See the inner exception details for information.. Duration: 00:00:49.7519745
Error details:
Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentNullException: Value cannot be null.
Parameter name: dn
at Unify.Framework.IO.DistinguishedName.op_Implicit(DistinguishedName dn)
at Unify.Product.IdentityBroker.Repository.EntityDistinguishedNameValueDataUtility`1.ConvertValueToString(DistinguishedNameValue value)
at Unify.Product.IdentityBroker.Repository.StringBasedValueDataUtilityBase`2.SetEntityValue(__EntityValueInsertRow dataValue, TValue value)
at Unify.Product.IdentityBroker.Repository.EntitySingleValueDataUtilityBase`2.CreateEntityValue(TEntityKey key, IValue value, IEntityCollectionKeyUtility`1 collectionKeyUtility, EntityDataSet set, __EntityInsertRow row, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.ConvertEntityValueToDataValue(KeyValuePair`2 entityValueAndKey, __EntityInsertRow row, EntityDataSet entityDataSet, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.<>c__DisplayClass33_0.b__0(KeyValuePair`2 entityValueAndKey)
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.d__17`2.MoveNext()
at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.InsertItems(ISet`1 addedItems, EntityDataContext sourceContext, SqlConnection connection)
at Unify.Framework.Data.LinqContextConversionBase`4.SubmitChanges()
at Unify.Product.IdentityBroker.SaveChangedEntitiesTransformationUnit.Transform(IDictionaryTwoPassDifferenceReport`4 input)
at Unify.Product.IdentityBroker.ConnectorEntityChangeProcessor.ProcessEntities(IEnumerable`1 connectorEntities, IEnumerable`1 repositoryEntities, IEntityChangesReportGenerator`2 reportGenerator)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_1.b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()
--- End of inner exception stack trace ---
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.WaitForCompletedThreads()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPool[T](IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.b__0()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)
Thanks for the update Hayden. I was just about to respond - it seems like there was a 'not-quite-null' value trying to be parsed into a DN field, which then when Broker was trying to store it in the entity context couldn't grab a valid string value to actually store. Some types in Broker, including the DN type, will treat an empty value differently to a null value - so if anything other than null is seen, it will attempt to convert (and in this case, fail).
Reflect change entities to adapter errors about duplicate entries when there are are no duplicate key values in the connector.
I've been having issues on particular PowerShell connectors/adapters in UNIFYBroker where reflecting change entities to the adapter is complaining about duplicate entries when there are are no duplicate key values in the connector.
The schema setup between the connectors and adapters is an ID key in the connector that is then used within the adapter as the DN. So it is a very simple DN template. E.g:
Name Type Key Read-only Required
AccountName String True True True
Distinguished Name Template CN=[AccountName]
The issues are generally fixed by clearing and repopulating the whole adapter, which is not a repeatable solution since it happens on a weekly basis, sometimes more often.
These errors also don't seem to happen after an obvious failures on the connector side, which is what I have previously attributed these issues to. All these connectors have deletion thresholds setup of at least 50%.
Its like broker seems to get itself tied up even though the schedules in the environment have been reduced to the point where only 1 operation is interacting within and with broker at a time.
SQL maintenance is also performed frequently and the SQL instance has plenty of resources allocated.
Version details: v5.3.2 Revision #0
Any help would be appreciated as this has been a long ongoing issue that I've seen across multiple environments.
Adapter
Adapter eb42757f-2f23-4228-928e-993942b0c050 page errored on page reflection. Duration: 00:00:21.5551444. Error: Unify.Framework.UnifyDataException: Duplicate DNs detected on adapter eb42757f-2f23-4228-928e-993942b0c050. Reflection failed. Duplicate DNs: CN=<obfuscated name>,OU=sIAMGroups,DC=IdentityBroker, CN=<obfuscated name 2>,OU=sIAMGroups,DC=IdentityBroker.
at Unify.Product.IdentityBroker.DuplicateDnDetector.DetectDuplicateDns(IDictionaryTwoPassDifferenceReport`4 report)
at Unify.Product.IdentityBroker.Adapter.ReflectChangesInner()
at Unify.Product.IdentityBroker.Adapter.ReflectChanges()
at Unify.Product.IdentityBroker.AdapterAuditingDecorator.ReflectChanges()
at Unify.Product.IdentityBroker.AdapterNotifierDecorator.ReflectChanges()
at Unify.Product.IdentityBroker.ReflectAdapterOnChangeDueJob.b__9_0(IOperationalAdapter adapter).
Error details:
Unify.Framework.UnifyDataException: Duplicate DNs detected on adapter eb42757f-2f23-4228-928e-993942b0c050. Reflection failed. Duplicate DNs: CN=<obfuscated name>,OU=sIAMGroups,DC=IdentityBroker, CN=<obfuscated name 2>,OU=sIAMGroups,DC=IdentityBroker.
at Unify.Product.IdentityBroker.DuplicateDnDetector.DetectDuplicateDns(IDictionaryTwoPassDifferenceReport`4 report)
at Unify.Product.IdentityBroker.Adapter.ReflectChangesInner()
at Unify.Product.IdentityBroker.Adapter.ReflectChanges()
at Unify.Product.IdentityBroker.AdapterAuditingDecorator.ReflectChanges()
at Unify.Product.IdentityBroker.AdapterNotifierDecorator.ReflectChanges()
at Unify.Product.IdentityBroker.ReflectAdapterOnChangeDueJob.b__9_0(IOperationalAdapter adapter)
Adapter Request to reflect change entities of the adapter.
Request to reflect change entities of the eMinerva Student: Groups (eb42757f-2f23-4228-928e-993942b0c050) adapter errored with message: Duplicate DNs detected on adapter eb42757f-2f23-4228-928e-993942b0c050. Reflection failed. Duplicate DNs: CN=<obfuscated name>,OU=sIAMGroups,DC=IdentityBroker, CN=<obfuscated name 2>,OU=sIAMGroups,DC=IdentityBroker.. Duration: 00:00:59.9516712
Error details:
Unify.Framework.UnifyDataException: Duplicate DNs detected on adapter eb42757f-2f23-4228-928e-993942b0c050. Reflection failed. Duplicate DNs: CN=<obfuscated name>,OU=sIAMGroups,DC=IdentityBroker, CN=<obfuscated name 2>,OU=sIAMGroups,DC=IdentityBroker.
at Unify.Product.IdentityBroker.DuplicateDnDetector.DetectDuplicateDns(IDictionaryTwoPassDifferenceReport`4 report)
at Unify.Product.IdentityBroker.Adapter.ReflectChangesInner()
at Unify.Product.IdentityBroker.Adapter.ReflectChanges()
at Unify.Product.IdentityBroker.AdapterAuditingDecorator.ReflectChanges()
at Unify.Product.IdentityBroker.AdapterNotifierDecorator.ReflectChanges()
at Unify.Product.IdentityBroker.ReflectAdapterOnChangeDueJob.b__9_0(IOperationalAdapter adapter)
Closed as no further information provided.
UNIFYConnect update threshold/safety catch
A UNIFYConnect customer has requested the ability to implement a safety catch feature to stop updates if they are over a certain threshold. I know that Broker/BrokerPlus has a safety catch feature for entity deletion thresholds, but does anything currently exist for updates? Ideally this would be a check that if the number of changes on a link are over X amount it stops the sync, disables the schedules for that link, and then throws an error in the logs to be picked up via the monitoring/alerting.
The only topic I could find on this at the moment is Safety Catch Feature / UNIFYBroker Forum / UNIFY Solutions.
Attribute changes don't trigger pending sync changes on a link if they are processed in a powershell task on the link
I have noticed that if a link has a PowerShell task on an outgoing sync task that modifies and sets attributes in the target, then a change flows through to only that attribute modified in the PowerShell task, then the link doesn't pick it up as pending sync change. Only running a baseline sync on the link would trigger the PowerShelltask to change/modify that attribute.
Discussion was had with Matt Davis on the possibility of using the Register-Contribution function on a link for the attributes that are being modified through the PowerShell task to then recognise changes for those entities. However it is unsure whether the Register-Contribution function is available for links or not. As an alternative, the PowerShell script can be used on the adapter in a reverse transform, which triggers on outgoing sync changes from a link to a connector. Then the contributing attributes can be sync as direct mappings in the link.
This has been implemented and is available in the release of UNIFYConnect V6, which will be made available shortly.
"Process unmapped changes" can be used to resolve this in the new software version, which will trigger changes for all entity fields rather than just those with direct mappings.
Alternatively add a direct mapping for the field and override with powershell sync logic.
Link Baseline sync still able to be run from Links page when link is disabled.
I have noticed that under normal operation, a baseline synchronization task on a link cannot be executed while the link is disabled. In the link UI, the option to run a baseline sync only becomes visible when the link is enabled. However I have found that on the Links page (that lists the links in the solution) selecting a disabled link and running a baseline sync through the Actions button at the top of the page still executes the baseline sync on the disabled link. Not sure if this is expected behaviour or is a bug.
Screenshot #1: Running a baseline sync task on a disabled link through the Actions button:
Screenshot #2: The baseline sync still executed even though the link is disabled:
This has been implemented and is available in the release of UNIFYConnect V6, which will be made available shortly.
Clarification of the Register-Contribution function
Hi,
Just seeking some clarification about how the "Register-Contribution" functions on PowerShell Schema transformations and what scenarios this applies to.
For example, if I have 2 fields Field1 and Field2 and apply the function like: "Register-Contribution Field1 Field2". Does this mean that a change to Field1 will trigger reprocessing changes of any transformations for Field2?
Also, would I be correct in assuming that this can be used to help process Time Offset Flag transformations where there may not necessarily be a change to retrigger evaluation of the flag?
Thanks,
Liam
Hey Liam,
There's a good explanation on this ticket which might help: https://unifyvoice.userecho.com/communities/6/topics/5330-time-offset-flag-not-re-evaluated-when-current-time-passes-source-field-timestamp
But essentially Field1 is a field you're creating with your powershell schema, and Field2 is an existing field which is used in calculating the result of Field1. You can call Register-Contribution multiple times on the same field, if more than one field contributes to it (see the detail at the bottom of this response)
The main use case for this is to allow automatic change detection through powershell transformations. Prior to this feature, the Broker change detection engine didn't know what the powershell script was doing - which fields were being used, and which fields were being output. Therefore, it had no way of knowing whether a change for a field should result in a change for another field.
So if you're using time offset flag transformations, where the input to your transformation is the result of a powershell script, then yes - this feature is the main use case. If you're using a time offset flag transformation with normal field mappings, then this transformation should already generate future dated changes for reprocessing even if there's no other adapter changes to trigger this.
It was something added as a patch in 5.3, and we've yet to do a documentation pass to add updated doco for it (we've got a draft ready for 6.0)
Some more technical detail on the contribution specifically which might help:
The signature of Register-Contributionis:
Register-Contribution -fieldName <string> -contributingField <string>
where fieldName is the name of the new field created with New-Field, and contributingField is the name of a pre-existing field that contributes to the new field in some way. This can be called multiple times for new fields that are contributed to from multiple pre-existing fields.
For example:
New-Field 'convertedValue' 'int'; New-Field 'joinedTS' 'timestamp'; Register-Contribution 'convertedValue' 'origValue'; Register-Contribution 'joinedTS' 'date'; Register-Contribution 'joinedTS' 'time';
Note that for most, normal usage of the PowerShell conenctor it won't be necessary to register field contributions. Time Offset Flag and Business Day Offset Flag transformations which use generated fields are the main reason why this would be needed.
Jobs Stuck Processing
Hi,
We have observed jobs such as Connector imports and Link synchronization will occassionally get stuck in a processing loop and not complete. This causes a block in operations as Broker cannot import or synchronize new data. To clear the process a restart has to be performed. Attempting to cancel the job does not have any impact.
This happens intermittently and doesn't appear to have a consistent way of reproducing the issue. I understand this makes it difficult to troubleshoot the issue, so is there other possibilities for a solution we could explore? For example, could there be a timeout introduced so that the job is killed if it runs over a period of time without closing?
Let me know your thoughts and feedback.
Thanks,
Liam
This has been resolved and is available in the release of UNIFYConnect V6, which will be made available shortly.
Powershell tasks can now have a timeout applied to them, which will force the cancellation of any running scripts after a given timeout which can ensure the scripts actually exit rather than getting stuck.
Link Synchronization not triggering
In UNIFYConnect test environment when attempting to perform a baseline synchronisation or delta changes synchronisation, the buttons seemingly do nothing when attempting to trigger on the link. The link is between a CSV connector/adapter and a locker with about ~10k entities. Nothing else is currently running, all other link schedules are disabled and no connectors are importing, nor are any change reflect options are running.
When I click a button to sync the page refreshes like it has executed but nothing else happens. Nothing appears under the "Recent Jobs" section of the link page and it logs 2 messages in the log:
23/Jan/2024 23:29:28
Information
Link Request to manually queue a baseline synchronization job on link started.
Request to manually queue a baseline synchronization job on link Managed User > AD User started.
23/Jan/2024 23:29:28
Information
Link Request to manually queue a baseline synchronization job on link completed.
Request to manually queue a baseline synchronization job on link 'Managed User > AD User' completed. Duration: 00:00:00.0310830
Is there a way I can see what is stopping the sync operation from executing? Let me know if you need more information.
Thank you
This has been implemented and is available in the release of UNIFYConnect V6, which will be made available shortly.
Messages are shown for both sync prep and sync execution, and the preparation task now performs significantly better so shouldn't sit for as long.
Customer support service by UserEcho
