Hello,

I have been tracking this issue for a few weeks within an environment. And what I have been noticing is the performance of the UNIFYBroker service deteriorating over time. While there is an existing work around, it would be good to find the root cause of this, whether it be a poorly configured item somewhere or an underlying issue.

Currently in the environment we notice the service become slower, at a minimum, over the course of about a week. It will gradually take sets of operation lists longer and longer to run. More specifically this will be worse on Monday after all the Full Baseline operations run over the weekend. And then gradually get even worse from there. The only correlation I can see so far is during the times it is running slow, is the service will be using upward of 5GB of ram, even getting to over 10GB if left unattended for longer than a week. It will retain these high levels of memory usage even while no operations are running. The only way to resolve the issue, is to restart the service, which currently happens on a Monday every week.

The environment is quite large with a Broker instance that manages over 1 million entities. However the server (both local and DB) has the specifications to deal with the load. Please see these below. It is also worth noting here also that to eliminate concurrency issues, the scheduling is setup to run everything sequentially (i.e UNIFYNow will step through each operation one at a time and no two operations will run at the same time both in MIM and UNIFYBroker).

Broker Server Specs:

CPU: 16 cores

Memory: 32GB

DB Server Specs:

CPU: 12 cores

Memory: 48GB

Although I haven't gathered this information for previous weeks, I have noticed some strange occurrences this morning and have documented them below:

• Old LDAP Gateway connection that has not been closed:
• 
• Large number of SQL Connection to the UNIFYBroker DB:
• 
• High service memory usage while nothing is running (as mentioned before)
• 

I will attach the logs and additional information below. Also in the logs below I have included information on the DB connections both before and after recycle the LDAP Gateway. But also note that recycling the LDAP Gateway also had no effect on the memory usage of the service. Let me know if there was anything else I can do to assist.

UNIFYBroker: v5.3.1

Please see attached Broker configuration.  The "MIM LMS Group Users" connector generates records for groups, with a multivalued field "PersonNumbers" that is then used by the "MIM LMS Person" adapter in a Foreign Multivalued Group Transformation to generate the DNs of the groups each user is a member of.  When a group has no members, running an Import All on the connector causes a "Value cannot be null" error to be logged.  If I change the source data to not include any groups with no PersonNumbers data then the error does not occur

As a workaround, I've inserted a dummy value into blank PersonNumbers field values as shown here:

I tried to replicate this issue in a simpler Broker instance, but I could not sorry.

After changing the Required flag for a field in my PowerShell schema script, I re-ran Request Schema and the UI didn't show any fields as changed.  After I Merged Changes the field was not updated, and I had to update the Required flag manually.

It seems a reasonable expectation that a change of any field attribute should be flagged by the UI and applied when Merge is clicked.

I am trying to configure an export flow in MIM for a DN field in UNIFYBroker, and I see this error:

However, when I use LDP.exe to connect to UNIFYBroker via the LDAP gateway I can Add or Replace a DN value in that field successfully, so the limitation appears to be in the UNIFYBroker ECMA2 DLL rather than in UNIFYBroker itself.

Could you please advise if this is the expected behaviour?  If it is then could you comment on why the limitation exists and advise what we should do as a workaround?

I added a Foreign Multivalue Group transform to an Adapter and the schema rows no longer showed.  When I then enabled the Adapter an error appeared: "Object reference not set to an instance of an object at Unify.Product.IdentityBroker.TemplateDistinguishedNameParser.ParseTemplate".

Connector "Owners":

Connector "Things":

Adapter:

Here is the transform (against the Owners connector):

When I attempt to enable the Adapter this error appears:

If I remove the Transform then the Adapter's schema re-appears:

If I attempt to edit the Transform an error appears:

Logs & Extensibility attached to next comment.

We've got a site where the adapter entities have been cleared and on a generate changes only one or two thousand entities get processed with one time the following error occurring:

"Transaction (Process ID 124) was deadlocked on lock resources with
another process and has been chosen as the deadlock victim. Rerun the
transaction.."

It looks like since the connector schedules haven't been disabled that they're running imports that are causing the database to deadlock, aborting the reflection process. I had a look through the documentation in the knowledge base but I couldn't find anything that confirms my suspicion.

What I'm looking for is confirmation that UNIFYBroker does hit a deadlock when a reflection and an import occur at the same on the same connector so I can advise the client on what to do in the future to avoid this (i.e., disable the schedules when doing a re-population).

The UNIFYBroker API returns null for SCIM gateways:

This is not the case for other gateway types, e.g. LDAP (example above).

Hello,

Not sure if this has been reported before, but couldn't see anything on my quick look. I am currently in an environment running Broker v5.3.1 Revision #4, and am experiencing what appears to be a UI bug where randomly the web page will begin to display "Service Unavailable" all over the page.

After several seconds to a minute or two, the page will then flick back to normal and continue in this pattern. Some days it will happen more often than others, with no increase in job frequency. Occasionally a service restart will improve the frequency of it occurring, however it will slowly return over time.

I've investigated a bit and it appears to not noticeably affect the current running jobs in UNIFYBroker, nor importing running from Broker into MIM. I have also see it happens more frequently when jobs are running in Broker, but not attributed to any one job in particular. There are also no errors in the Event Log, however there are some in the UNIFYBroker log, though they don't appear to be UI specific, nor can I say I have seen these on every occasion this happens.

The error I'm referring to in particular is:

"Unable to raise complete notification:

System.ApplicationException: This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4)
at System.Threading.ReaderWriterLock.AcquireWriterLockInternal(Int32 millisecondsTimeout)
at System.Threading.ReaderWriterLock.AcquireWriterLock(TimeSpan timeout)
at Unify.Framework.Collections.ThreadsafeObjectQueueBase`2.Push(TItem item)
at Unify.Framework.Notification.TaskNotifier`1.Dispose()"

If you need any more information just let me know.

Thanks

Using swagger I see the following error when I call the Link/PowerShell/UpdateProvisioningTask function:

The full response body is:

{
"Message": "An error has occurred.",
"ExceptionMessage": "Index was out of range. Must be non-negative and less than the size of the collection.\r\nParameter name: index",
"ExceptionType": "System.ArgumentOutOfRangeException",
"StackTrace": " at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)\r\n at System.Collections.Generic.List`1.set_Item(Int32 index, T value)\r\n at Unify.Product.Plus.LinkEngine.<>c__DisplayClass92_0.<UpdateProvisioningTask>b__0()\r\n at Unify.Product.Plus.LinkEngine.<>c__DisplayClass161_0.<ConfigurationChange>b__0()\r\n at Unify.Framework.ExtensionMethods.WaitOnMutex(Mutex mutex, Action work)\r\n at Unify.Product.Plus.LinkEngine.UpdateProvisioningTask(Guid linkId, Boolean incoming, ProvisioningStep step, IProvisioningTaskInformation taskInformation)\r\n at Unify.Product.Plus.LinkEngineAuditingDecorator.UpdateProvisioningTask(Guid linkId, Boolean incoming, ProvisioningStep step, IProvisioningTaskInformation taskInformation)\r\n at Unify.Product.Plus.LinkEngineNotifierDecorator.<>c__DisplayClass50_0.<UpdateProvisioningTask>b__0()\r\n at Unify.Framework.Notification.NotifierDecoratorBase.Notify(ITaskNotificationFactory notificationFactory, Action action)\r\n at Unify.Product.Plus.LinkControllerBase.InnerUpdateProvisioningTask[TExtended](Guid linkId, Boolean incoming, ProvisioningStep step, Guid taskId, IProvisioningTaskApiInformation`1 taskInformation, XElement extended)\r\n at Unify.Product.Plus.LinkController.UpdatePowerShellProvisioningTask(Guid linkId, Boolean incoming, ProvisioningStep step, Guid taskId, PowerShellProvisioningTaskApiInformation taskInformation)\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClassc.<GetExecutor>b__6(Object instance, Object[] methodParameters)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()"
}

I will attach my configuration extensibility files to the next comment.

JSON objects returned by a REST API call to Link/GetProvisioningInformation contains task informations with two nested Extended attributes.  As a comparison point, only one Extended attribute may be specified in the task information object when calling Link/PowerShell/AddProvisioningTask, so this seems inconsistent and may be wrong.

Also, as a minor point the Example value for this call is not particularly helpful - it looks like an XML data structure converted to JSON and doesn't reflect the structure of data the call typically returns on a live system.