Hi All,

Historically we have had this issue prior to upgrading the latest version of UNIFYBroker (on v5.0.0). However we are now on version 5.3.2, and didn't seen the errors for some time, though now they appear to be occurring again. The errors so far have only been seen occurring on every DIDS after a failing full import operations. As mentioned before UNIFYBroker is on the latest version and FIM this user the latest version on the Unify extension (v5.3.0).

The FIM operation currently has a page size of 1000 and an operation timeout of 1500 seconds. So it appears to get stuck for quite some time.

The UNIFYBroker Log doesnt have anything on the exact time of failure but it does appear to be surrounded by Timeout errors:

20191215,17:43:12,UNIFYBroker,Void AcquireReaderLockInternal(Int32),Warning,"mscorlib:
System.ApplicationException: This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4)
at System.Threading.ReaderWriterLock.AcquireReaderLockInternal(Int32 millisecondsTimeout)
at System.Threading.ReaderWriterLock.AcquireReaderLock(TimeSpan timeout)
at Unify.Framework.Data.LinqQueryConversionProvider`5.ExecuteMethodCallExpression[TResult](MethodCallExpression methodCallExpression)
at Unify.Framework.Data.LinqQueryConversionProvider`5.Execute[TResult](Expression expression)
at System.Linq.Queryable.Count[TSource](IQueryable`1 source)
at Unify.Product.IdentityBroker.AdapterStatisticsEngine.GetEntityCount(Guid adapterId)",Normal

Let me what other information you need, and I'll be happy to provide it.

Thanks

I am using the following PS Adapter transform to generate a "hrStatus" value for EVERY record:

foreach ($entity in $entities) {
    [string]$hrStatus = "Active"
    if (!($entity["uid"]) -or ($entity["uid"].Value -notlike "A*") -or ($entity["uid"].Value -like "*_*")) {
        $hrStatus = "Inactive"
    }
    $entity["hrStatus"] = $hrStatus
}

I declare the schema for the hrStatus property as follows:

New-Field 'hrStatus' 'string' $false $true $true;

I am then setting the DN to use this property as a next-level OU below OU=Employees.

However when I enable the Adapter I get the error below - does this mean that the REQUIRED property is being ignore?

System.Exception: Swagger Exception could not be parsed. SE response code: 500; SE response text: {"Message":"An error has occurred.","ExceptionMessage":"Error in adapter DotEE Employee distinguished name configuration: The DN component part 'OU=[hrStatus]' could not be executed as the field hrStatus is not required. An empty field would result in a DN of 'OU='.","ExceptionType":"Unify.Framework.UnifyConfigurationException","StackTrace":"   at Unify.Product.IdentityBroker.AdapterEngine.ValidateAdapterForEnabling(IOperationalAdapter adapter)\r\n   at Unify.Product.IdentityBroker.AdapterEngine.<>c__DisplayClass54_0.<EnableAdapter>b__0()\r\n   at Unify.Product.IdentityBroker.AdapterEngine.<>c__DisplayClass145_0.<ConfigurationChange>b__0()\r\n   at Unify.Framework.ExtensionMethods.WaitOnMutex(Mutex mutex, Action work)\r\n   at Unify.Product.IdentityBroker.AdapterEngineAuditingDecorator.EnableAdapter(Guid adapterId)\r\n   at Unify.Product.IdentityBroker.AdapterEngineNotifierDecorator.<>c__DisplayClass19_0.<EnableAdapter>b__0()\r\n   at Unify.Framework.Notification.NotifierDecoratorBase.Notify(ITaskNotificationFactory notificationFactory, Action action)\r\n   at lambda_method(Closure , Object , Object[] )\r\n   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClassc.<GetExecutor>b__6(Object instance, Object[] methodParameters)\r\n   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()","InnerException":{"Message":"An error has occurred.","ExceptionMessage":"The DN component part 'OU=[hrStatus]' could not be executed as the field hrStatus is not required. An empty field would result in a DN of 'OU='.","ExceptionType":"Unify.Framework.UnifyEngineException","StackTrace":"   at Unify.Product.IdentityBroker.FieldTemplateDistinguishedNameComponentExecutor`2.Validate(IEntitySchema schema)\r\n   at Unify.Product.IdentityBroker.TemplateDistinguishedNameExecutor`2.Validate(IEntitySchema schema)\r\n   at Unify.Product.IdentityBroker.AdapterEngine.ValidateAdapterForEnabling(IOperationalAdapter adapter)"}}; ---> Unify.Framework.Client.SwaggerException: The HTTP status code of the response was not expected (500).
   at Unify.Connect.Web.Client.AdapterClient.<ToggleAdapterEnabledAsync>d__117.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Unify.Connect.Web.Client.ProfiledAdapterClient.<ToggleAdapterEnabledAsync>d__124.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Unify.Framework.Web.UnifyController.<HandleRemoteException>d__42.MoveNext()
   --- End of inner exception stack trace ---
   at Unify.Framework.Web.UnifyController.InnerHandleSwaggerExceptionForApiCall(SwaggerException se, String messageTemplate, Action`1 handleMessage, Action`1 handleExtended)
   at Unify.Framework.Web.UnifyController.HandleSwaggerExceptionForApiCall(SwaggerException se, String messageTemplate)
   at Unify.Framework.Web.UnifyController.<HandleRemoteException>d__42.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Unify.Connect.Web.AdapterController.<ToggleEnableState>d__74.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Unify.Connect.Web.AdapterController.<EnableAdapter>d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at lambda_method(Closure , Task )
   at System.Web.Mvc.Async.TaskAsyncActionDescriptor.EndExecute(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass34.<BeginInvokeAsynchronousActionMethod>b__33(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3c()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass45.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3e()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass1e.<>c__DisplayClass28.<BeginInvokeAction>b__19()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass1e.<BeginInvokeAction>b__1b(IAsyncResult asyncResult)

Hi Guys,

We've had a couple of entities today reported not provisioning into FIM. So I did some investigation and I found that the entities have come into UNIFYBroker through a DB Connector and then added to the adapter. However I cannot see a change log entry in the DB to then flow in via a delta in FIM.

So far I have check the change log table as mentioned and the UNIFYBroker logs. I cannot see any errors around the time the record was created in the adapter nor can I see any errors referencing the connector/adapter combo.

Currently we are on the latest version of Broker v5.3.2.

As a workaround we are doing a Full Import to bring it into FIM but it takes quite a while as its a large MA, so it would be good to get to the bottom of this. If you need any more information or need me to do some more digging I'm be happy to help.

Thanks

The connector is able to push data to AD, but throws an error when performing a change/all import from AD.

Has anyone run into this issue on imports?

20191210,04:34:48,UNIFY Identity Broker,Change detection engine,Information,"Change detection engine import all items started.
Change detection engine import all items for connector Active Directory Connector started.",Normal
20191210,04:34:54,UNIFY Identity Broker,Change detection engine,Error,"Change detection engine import all items failed.
Change detection engine import all items for connector Active Directory Connector failed with reason One or more errors occurred.. Duration: 00:00:05.5141233
Error details:
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Unify.Connectors.AD.ADConnector.TransformEntry(ADAgent agent, SearchResultEntry searchResultEntry, Int64& uSNChangedToken)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at Unify.Framework.Collections.EnumerableExtensions.<actiononlast>d__10`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.AuditReadingAsyncConnectorDecorator.<>c__DisplayClass1_0.<getallentitiesasync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Unify.Framework.Auditing.AuditingExtensions.<>c__DisplayClass7_0`1.<createandsendlogentryasync>b__0(Task`1 t)
   at Unify.Framework.Auditing.AuditingExtensions.<>c__DisplayClass5_0`1.<taskcontinuewithexceptionpassthough>b__0(Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Unify.Product.IdentityBroker.EventNotifierReadingAsyncConnectorDecoratorBase`1.<getallentitiesasync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Unify.Product.IdentityBroker.ChangeDetectionImportAllAsyncJob.<importallchangeprocess>d__6.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Unify.Product.IdentityBroker.ChangeDetectionImportAllAsyncJob.RunBase()
   at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
   at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.<run>b__0()
   at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Unify.Connectors.AD.ADConnector.TransformEntry(ADAgent agent, SearchResultEntry searchResultEntry, Int64& uSNChangedToken)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at Unify.Framework.Collections.EnumerableExtensions.<actiononlast>d__10`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.AuditReadingAsyncConnectorDecorator.<>c__DisplayClass1_0.<getallentitiesasync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Unify.Framework.Auditing.AuditingExtensions.<>c__DisplayClass7_0`1.<createandsendlogentryasync>b__0(Task`1 t)
   at Unify.Framework.Auditing.AuditingExtensions.<>c__DisplayClass5_0`1.<taskcontinuewithexceptionpassthough>b__0(Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Unify.Product.IdentityBroker.EventNotifierReadingAsyncConnectorDecoratorBase`1.<getallentitiesasync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Unify.Product.IdentityBroker.ChangeDetectionImportAllAsyncJob.<importallchangeprocess>d__6.MoveNext()
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.
   at Unify.Connectors.AD.ADConnector.TransformEntry(ADAgent agent, SearchResultEntry searchResultEntry, Int64& uSNChangedToken)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at Unify.Framework.Collections.EnumerableExtensions.<actiononlast>d__10`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Product.IdentityBroker.AuditReadingAsyncConnectorDecorator.<>c__DisplayClass1_0.<getallentitiesasync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()<---
<---
",Normal
</getallentitiesasync></actiononlast></importallchangeprocess></getallentitiesasync></taskcontinuewithexceptionpassthough></createandsendlogentryasync></getallentitiesasync></actiononlast></run></importallchangeprocess></getallentitiesasync></taskcontinuewithexceptionpassthough></createandsendlogentryasync></getallentitiesasync></actiononlast>

When switching from a PowerShell to C# connector for the same schema, the following error occurred when attempting to refresh the adapter (after Generate Changes):

Request to reflect change entities of the adapter.
Request to reflect change entities of the DotEE Company (1425b9d9-bdd2-4786-81ae-8c09272a0750) adapter errored with message: An error occurred retrieving the distinguished name component for field 'companyCode' of type CN. See inner exception for details.. Duration: 00:00:00.0937585
Error details:
Unify.Framework.UnifyDataException: An error occurred retrieving the distinguished name component for field 'companyCode' of type CN. See inner exception for details. ---> Unify.Framework.Collections.GroupedNameValueCollectionMissingFieldException: The entity does not contain a value for the companyCode field.
   at Unify.Product.IdentityBroker.EntityBase`3.GetValueEntry(TKey key)
   at Unify.Product.IdentityBroker.FieldTemplateDistinguishedNameComponentExecutor`2.Get(TEntity entity)
   --- End of inner exception stack trace ---
   at Unify.Product.IdentityBroker.FieldTemplateDistinguishedNameComponentExecutor`2.Get(TEntity entity)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Unify.Framework.IO.DistinguishedName..ctor(IEnumerable`1 components)
   at Unify.Product.IdentityBroker.TemplateDistinguishedNameExecutor`2.DistinguishedName(TEntity entity)
   at Unify.Product.IdentityBroker.Adapter.<>c__DisplayClass120_0.<ConvertPageAndUpdateContainers>b__1(IEntity entity)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at Unify.Product.IdentityBroker.Adapter.ConvertPageAndUpdateContainers(IEntity[] entities, Boolean updateContainers)
   at Unify.Product.IdentityBroker.Adapter.ReflectChangesInner()
   at Unify.Product.IdentityBroker.Adapter.ReflectChanges()
   at Unify.Product.IdentityBroker.AdapterAuditingDecorator.ReflectChanges()
   at Unify.Product.IdentityBroker.AdapterNotifierDecorator.ReflectChanges()
   at Unify.Product.IdentityBroker.ReflectAdapterOnChangeDueJob.<RunBase>b__9_0(IOperationalAdapter adapter)

In troubleshooting SAP connectivity I am repeatedly making changes to the Unify.Service.Connect.exe.config file and restarting Broker - but being careful not to start until the existing process has completely stopped - which usually takes about 30 seconds in my environment.

I got tired of doing this manually and wrote the following simple script - which must be run from an Administrator PS session:

cls
[bool]$isRunning = [bool](Get-Process "Unify.Service.Connect" -ErrorAction SilentlyContinue)
if ($isRunning) {
    net stop service.connect
}
[int]$counter = 0
while ($isRunning) {
    $counter++
    Write-Host "Waiting [$counter] seconds for service to shut down ..."
    Start-Sleep 1
    $isRunning = [bool](Get-Process "Unify.Service.Connect" -ErrorAction SilentlyContinue)
}
net start service.connect

I hope others find this useful too!

MIM has around 940 pending export deletes for HPPhoneNumber attributes, and they are being processed successfully by UNIFYBroker and the Aderant Expert connector (i.e. no export errors).  However, during a subsequent Full Import by MIM that attribute is being restored to the non-NULL value, meaning that the next Full Sync results in each of those objects having a new pending export, ad infinitum.

Here's the PowerShell transform that was required (so it can be ported to C#):

# Truncate long fields

function Invoke-FieldTruncator {
param(
$Entity,
[string] $Field,
[int] $Length
)

$old = $Entity[$Field].Value
if ($old) {
$old = $old.ToString()
if ($old.Length -gt $Length) {
$new = $old.Substring(0, $Length)
$Logger.LogInformation("Truncated long $Field '$old' to '$new'")
$Entity[$Field] = $new
}
}
}

foreach ($entity in $entities) {
Invoke-FieldTruncator -Entity $entity -Field HPPhoneNumber -Length 17
Invoke-FieldTruncator -Entity $entity -Field NameSort -Length 30
}

Note that these are only the fields that had long data which needed truncation in the DEV environment.  When we move to UAT and PROD it is likely there will be other fields with long data that needs to be truncated.  A lot of time and debugging effort was expended by the consultant (me) to identify and remediate these two fields in DEV, and this will add to the time required for UAT and PROD deployments.  The extra time required will be a particular issue when we come to PROD as it will significantly increase the deployment time during which time the system will be down for the customer.

As a consequence, I suggest that all fields be truncated to their maximum database lengths, not just those listed in the DEV workaround above.

After a SQL timeout it appears the SQL connection to Aderant Expert remains with a SQL transaction that has not been disposed.  The error is:

20191203,23:22:24,UNIFYBroker,Connector,Warning,"Update entities to connector failed.
Update entities [Count:2071] to connector Global Aderant Expert Connector failed with reason The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements.. Duration: 00:01:01.0504286

The previous timeout error responsible for the undisposed transaction is:

20191203,23:16:16,UNIFYBroker,Connector,Warning,"Update entities to connector failed.
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
Update entities [Count:2071] to connector Global Aderant Expert Connector failed with reason Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.. Duration: 00:01:16.0479697

The workaround is to restart the UNIFYBroker service, to stop it reusing the bad SQL connection.

See also https://voice.unifysolutions.net/communities/6/topics/3995-aderant-expert-agent-ui-doesnt-save-changes-to-the-operation-timeout-parameter for more information about SQL operation timeouts on a Global Aderant Expert connector.

After changing and saving the "Operation Timeout" parameter on the Aderant Expert agent UI the value is not saved.  In the extensibility file, the value "PT0S" is always written.

As a workaround I have edited the extensibility file instead.