Identity Broker Forum

Welcome to the community forum for Identity Broker.

Browse the knowledge base, ask questions directly to the product group, or leverage the community to get answers. Leave ideas for new features and vote for the features or bug fixes you want most.

+1
Answered

Entity in IdB connector and adapter but does not exist in target directory

Carol Wapshere 3 years ago in PowerShell connector • updated by anonymous 3 years ago 3

IdB 5, Powershell connector, target system is RedHat LDAP.


There are three objects which exist as entities in the IdB connector and adapter but do not exist in LDAP. FIM is trying to update them and we're getting "Object does not exist" errors back from LDAP.


Connector Full Imports have been run. I turned on the verbose logging I'd added to the script which lists the DN of every object found by the Import script and these objects are not listed. I can't see any errors in the IdB log and the Full Import appears to have completed successfully.


So the question is, if they were not imported in a connector full import, shouldn't the entities have been removed from IdB?

Answer
anonymous 3 years ago

Looking at the logs shows that there were exported entities during the full import. The import logic is designed to not delete entities that are added whilst an import is occurring, as it has no way of knowing whether the end system is omitting the entry because it was deleted immediately or because it’s just not available yet for the import (e.g. snapshot or read copy/write copy style systems).

0
Answered

Import from connector failed with reason User Not Found. User: None

Tom Parker 2 weeks ago in PowerShell connector • updated by Beau Harrison 2 weeks ago 5

Hi is there any guidance on what this error means? I'm getting it on imports on a powershell connector. I assumed that it meant it's trying to commit an entity that's missing some attribute but the code seems to be correct. Snippet and error follow.

Only ID is a required attribute.

Relevant part of import.ps1


if ($msoluser.ImmutableID)

{

## Create or update IdB entity

$entity = $entities.Create()

$entity['ID'] = $msoluser.ImmutableID

$entity['UPN'] = $msoluser.UserPrincipalName

$entity['isLicensed'] = $msoluser.isLicensed

$entity['Alumni'] = $Alumni

$entity['UserStatus'] = $UserStatus

$entity['CheckLicense'] = $CheckLicense

$entity.Commit()

}

Error in IDB logs:

Import changes from connector Office 365 Student Connector failed with reason User Not Found.  User: none.. Duration: 00:45:45.8101253
Error details:
Microsoft.Online.Administration.Automation.MicrosoftOnlineException: User Not Found.  User: none.
   at Unify.Product.IdentityBroker.PowerShellConnector.d__30.MoveNext()
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at Unify.Framework.Collections.ActionOnExceptionEnumerator`1.MoveNext()
   at Unify.Framework.Collections.EnumerableExtensions.d__10`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.ChangeDetectionEntityPollJob.RunBase()
   at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
   at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass33_0.b__0()
   at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal
Answer
Beau Harrison 2 weeks ago

Hi Tom, based of the exception type it looks like an exception generated by the Microsoft code that's fetching a user.

Microsoft.Online.Administration.Automation.MicrosoftOnlineException: User Not Found.  User: none.

Using the PowerShell logger to add additional logging around keep parts of the script should help you confirm this and debug further.

0
Answered

Error occurs once per page "Cannot access destination table 'EntityValueOrigin'"

Tom Parker 3 months ago in PowerShell connector • updated 3 months ago 5

Once per page on export the following error occurs.There doesn't appear to be any impact from the error as the user is provisioned correctly by powershell and they also appear correctly in UNIFYBroker. They also appear in both the "adds" section of MIM and the "errors" section under "unexpected-error". The following stack trace appears in the Identity Broker logs.

We very recently upgraded from 5.0.3 but we are unsure if that's related.

UNIFYBroker Version: 5.1.0 Revision #2

MIM Version: 4.4.1749.0


System.InvalidOperationException: Cannot access destination table 'EntityValueOrigin'. ---> System.Data.SqlClient.SqlException: Invalid object name 'EntityValueOrigin'.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlBulkCopy.RunParser(BulkCopySimpleResultSet bulkCopyHandler)
at System.Data.SqlClient.SqlBulkCopy.CreateAndExecuteInitialQueryAsync(BulkCopySimpleResultSet& result)
at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource`1 source)
--- End of inner exception stack trace ---
at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource`1 source)
at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken)
at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken)
at System.Data.SqlClient.SqlBulkCopy.WriteToServer(DataTable table, DataRowState rowState)
at Unify.Product.IdentityBroker.EntityValueOriginContext.InsertItems(ISet`1 addedItems, EntityValueOriginDataContext sourceContext, SqlConnection connection)
at Unify.Framework.Data.LinqContextConversionBase`4.SubmitChanges()
at Unify.Product.IdentityBroker.OriginInformationProcessor.RunBase()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal

Answer
Curtis Lusmore 3 months ago

Please run the Database upgrade script, located in the Database sub-directory of the Identity Broker installation directory.

0
Answered

Powershell Connector continuing to run script after MIM says run is complete

Tom Parker 4 months ago in PowerShell connector • updated by Bob Bradley 4 months ago 4

It looks like a Powershell MA's script is continuing to run well after the MA in MIM says the run has been completed. Is this a known thing which happens or is intended or is it a bug?

UNIFY Broker Version: v5.1.0 Revision #2
MIM Version 4.4.17849.0

AddUser powershell code: D:\ADProvisioning.Powershell\UserAdd.ps1

Note: The code was previously "& D:\ADProvisioning.Powershell\UserAdd.ps1" but I changed it because I didn't understand the intent of running it as a separate process and to simplify the problem solving process. The issue still occurred when


Before running the export:

No log in file explorer yet for the AD Provisioning Export.

After running the export

All the errors are ma-extension-error, which may be genuine as the script I'm writing is still being worked on.

Here's the number of users steadily increasing

Here's the export log continuing to be updated after the run has been finished.



I can make the script I'm running available on request.

Answer
Curtis Lusmore 4 months ago

The issue turned out to be that the MIM Agent is timing out. Please see https://voice.unifysolutions.net/knowledge-bases/7/articles/3364-unifybrokermicrosoft-identity-manager-configuration for details on configuring the timeout. Please note in particular that bulk exports use only a single request per page (the Page Size setting), so you will need to either decrease the page size, increase the timeout, or improve the performance of the PowerShell script.

0
Under review

All objects reported as changed on import

Boyd Bostock 6 months ago in PowerShell connector • updated 2 weeks ago 8

When a Full Import on a Connecter is performed all entities are reported as having changed data every time the import is run. This is causing issues as processing time is longer than necessary and Adapter processing can queue up during peak times.

The Connector in question has a large number of attributes including several multivalued fields. All multivalued fields are sorted and uniqueness is enforced so I do not believe any attributes are changing.

I have cleared the Adapter and Connector and it still occurs, the same Connector and Adapter is run on another server and does not experience the same issue.

Is there a way to determine which attribute is being reported as changed?

0
Answered

PowerShell connector intermittently haning on Polling import

Carol Wapshere 6 months ago in PowerShell connector • updated by Beau Harrison 5 months ago 10

I have an intermittent problem with particular PowerShell connectors that intermittently hang on the Polling import - in that the connector displays as running the polling import for days, from the logs nothing is happening, and the only way to stop it is to restart the IDB service.

I have three connectors that connect to Exchange (two different Exchange environments), and we have seen the problem on all three connectors, in all three environments (dev, test, prod). I have other PowerShell connectors that do not have this problem. We have also never seen the problem on the Import All.

The three connectors run the same script, just with different parameters. I have added detailed logging for Polling runs and can't find a pattern - the log files stop at different places. Sometimes it's while collecting data from Exchange, but just as often it's after the script has closed the connection to Exchange and is looping through updating the entities in IDB.

Is there any way to enforce a timeout in the Powershell connector?

Answer
Curtis Lusmore 6 months ago

Hi Carol,

There isn't currently any way to enforce a timeout in the PowerShell connector. If the commands which hang don't have convenient timeout flags, you could try using Start-Job and Wait-Job.

0
Answered

The network path was not found

I suddenly have an error with a connector that is one of three identical Powershell connectors (same underlying scripts, just different parameters specifying target domain). The data gathering part of the script is working fine. The script is also getting through the entire entity creation loop (I have dropped detailed logs), but is then failing after that (ie at the end of the script) with 0 entities created.

This is the error reported in the IdB log:

Change detection engine import all items failed.
Change detection engine import all items for connector PowerShell HomeFolder Protected failed with reason One or more errors occurred.. Duration: 00:08:50.4965198
Error details:
System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The network path was not found
--- End of inner exception stack trace ---
at Unify.Product.IdentityBroker.PowerShellConnector.d__30.MoveNext()
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at Unify.Product.IdentityBroker.AuditReadingConnectorDecorator.GetAllEntities(IStoredValueCollection storedValues, CancellationToken cancellationToken)
at Unify.Product.IdentityBroker.EventNotifierReadingConnectorDecoratorBase`1.GetAllEntities(IStoredValueCollection storedValues, CancellationToken cancellationToken)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.b__0()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)
---> (Inner Exception #0) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #1) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #2) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #3) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #4) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #5) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #6) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #7) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #8) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---

---> (Inner Exception #9) System.ComponentModel.Win32Exception (0x80004005): The network path was not found<---
Answer
Matthew Davis 10 months ago

Thanks for the update Carol. It is strange that it's saving the error until the end of the script - we'll do some investigation to see if we can work out why that's happening.

0
Answered

An item with the same key has already been added

Carol Wapshere 12 months ago in PowerShell connector • updated by Beau Harrison 12 months ago 3

I have just upgraded IdB in TEST to 5.2, and migrated in the Connector and Adapter files from Dev. Dev was already on 5.2 and all connectors are working.

In TEST all of the new Connectors (names "PowerShell HomeFolder*" and "PowerShell MemberOf*") are failing with the error below. The "Powershell Exchange*" connectors work fine (though they already pre-existing in IdB 5.1 before I upgraded to 5.2).

The Connector config file is the same as the one I sent with the previous question. While the error looks very similar to that one it can't be the same - that was a duplicate schema mapping in the connector config, but Powershell connectors don't have schema mapping.

My Import scripts drop a full log of all entity values before running the $entity.Create ... $entity.Commit loop. There are no duplicate sAMAccountNames.

Note that when I did the IdB database upgrade in Test I removed the final lines from the script as told to do here. I don't seem to have had any problems with this in Dev, but thought it worth mentioning.


Change detection engine import all items failed.
Change detection engine import all items for connector PowerShell HomeFolder NMI failed with reason An error occurred while evaluating a task on a worker thread. See the inner exception details for information.. Duration: 00:01:37.5326976
Error details:
Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
at Unify.Product.IdentityBroker.Repository.EntityLinqQueryConverterUtilitiesBase`4.GetCollectionKeyData(TEntityKey key, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.EntitySingleValueDataUtilityBase`2.CreateEntityValue(TEntityKey key, IValue value, IEntityCollectionKeyUtility`1 collectionKeyUtility, EntityDataSet set, __EntityInsertRow row, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.ConvertEntityValueToDataValue(KeyValuePair`2 entityValueAndKey, __EntityInsertRow row, EntityDataSet entityDataSet, EntityDataContext sourceContext)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.<>c__DisplayClass31_0.<convertitemtovalues>b__0(KeyValuePair`2 entityValueAndKey)
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.<selectmanyiterator>d__16`2.MoveNext()
at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
at Unify.Product.IdentityBroker.Repository.KnownEntityContextBase`4.InsertItems(ISet`1 addedItems, EntityDataContext sourceContext, SqlConnection connection)
at Unify.Framework.Data.LinqContextConversionBase`4.SubmitChanges()
at Unify.Product.IdentityBroker.SaveChangedEntitiesTransformationUnit.Transform(IDictionaryTwoPassDifferenceReport`4 input)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetectionOnConnectorEntityPage(IEnumerable`1 connectorEntities, Int32& index, Int32 entitiesProcessedSoFar, IEntityChangesReportGenerator`2 reportGenerator, IHashSet`1 seenKeys)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.<>c__DisplayClass11_0.<performchangedetection>b__0(IEnumerable`1 page)
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.ThreadsafeItemEvaluator.Evaluate()
--- End of inner exception stack trace ---
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.CheckForException()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.<run>b__0()
at Unify.Framework.<span class="redactor-selection-marker" id="selection-marker-1"></span>AsynchronousJobExecutor.PerformJobCallback(Object state)
</run></performchangedetection></selectmanyiterator></convertitemtovalues>
Answer
Beau Harrison 12 months ago

For future reference, this issue is caused by entries in the CollectionKey table with the same Caption field value. The duplicate captions, produced by a defect in Identity Broker v5.1, cause exceptions to be thrown in several areas of the application after performing an upgrade to Identity Broker v5.2 which assume these values to be unique.

The simplest solution for this issue would be running the database clear script found in the <InstallDir>/Database directory. If this is not possible or desirable, attempt to run the script I provided below which clears the CollectionKey table of all unused entries and may resolve this issue. If the issue persists at this point a script or tool can be provided suitable to the specific environment to more directly correct the database state.

0
Answered

Access denied importing Boolean value

Carol Wapshere 1 year ago in PowerShell connector • updated 1 year ago 2

I have been struggling with an odd issue. I have a number of PowerShell connectors in my solution, and quite a few of them have Boolean attributes (in addition to String). I have created a new connector which is very similar to the existing ones, and uses very similar scripts. Here is what is happening:

- When I comment out my import step for the Boolean attribute (so it just imports the String attributes) the full import works

- When I add the Boolean attribute in the import fails with "Access Denied". I know the correct value is actually being generated as I'm dropping a debug log file which shows all the values it's trying to put into the entity.

- When I run a Polling import (where I can list account names in a text file and it just imports those) it works correctly, including bringing in the Boolean value. This is also weird as it's the same Powershell script - the differences are in the initial collection of data, but the part where it updates and commits the entities is identical for both full and polling imports.

I thought it might be something to do with $null values but have made sure that the script always sets a default value of $false. This is the error message - it is definitely happening during the loop where I go through the data I've collected committing the entities.

I've also upgraded IdB to 5.2, but no change from 5.1 on this issue.


Change detection engine import all items for connector PowerShell HomeFolder failed with reason Access is denied. Duration: 00:01:19.9160765

Error details:

System.UnauthorizedAccessException: Access is denied ---> System.ComponentModel.Win32Exception: Access is denied

   --- End of inner exception stack trace ---

   at Unify.Framework.Auditing.AuditingExtensions.<>c__DisplayClass4_0.<TaskContinueWithExceptionPassthough>b__0(Task t)

   at System.Threading.Tasks.Task.Execute()",Normal


Answer
Curtis Lusmore 1 year ago

Via phone call, the problem was that the line which was being commented out includes a call to Test-Path, and we believe the permissions error is happening there.

0
Answered

Accessing a multivalue String attribute

Carol Wapshere 1 year ago in PowerShell connector • updated by Curtis Lusmore 1 year ago 3

Been searching the doco and other Voice questions but I can't find this one - how do I access a multivalue string attribute in a PowerShell connector script?

I have tried $entity["attrib"].Values, but that doesn't work.

Using $entity["attrib"].Value seems to return a long string with all the values joined by semi-colons - at least that's how it gets written to the logger. I tried splitting on semi-colon but got "Method invocation failed because [Unify.Framework.StringValue] does not contain a method named "Split"."

It would be helpful if there were examples for different data types including multi-value on this page: https://voice.unifysolutions.net/knowledge-bases/7/articles/2912-powershell-connector-entities.

Answer
Curtis Lusmore 1 year ago

Hi Carol, Thanks for raising this.

When you use the indexing operator on an entity, you get an IValue object, which contains a Value member containing the raw .Net value for that field.

In the case of a multi-value, the raw .Net value will be a List<IValue> containing the individual values - you may then need to access the Value member of each of those to access the individual raw .Net values.

As an example, consider the following script, where MV is a multi-valued integer field.

foreach ($entity in $components.InputEntities) {
    $values = $entity['MV'].Value; # List of IntegerValue
    $logger.LogWarning($values.GetType()); # Logs System.Collections.Generic.List[Unify.Framework.IntegerValue]
    $count = 0;
    $values | % { $logger.LogWarning($_.GetType()); $count += $_.Value } # Logs Unify.Framework.IntegerValue and sums raw .Net integer values into $count
}

Please let me know if this example clarifies this for you, and I'll update the documentation as you suggest.