0
Under review

Deteriorating operation performance over time

Hayden Gray 1 month ago in UNIFYBroker Service updated 5 days ago 2

Hello,

I have been tracking this issue for a few weeks within an environment. And what I have been noticing is the performance of the UNIFYBroker service deteriorating over time. While there is an existing work around, it would be good to find the root cause of this, whether it be a poorly configured item somewhere or an underlying issue.

Currently in the environment we notice the service become slower, at a minimum, over the course of about a week. It will gradually take sets of operation lists longer and longer to run. More specifically this will be worse on Monday after all the Full Baseline operations run over the weekend. And then gradually get even worse from there. The only correlation I can see so far is during the times it is running slow, is the service will be using upward of 5GB of ram, even getting to over 10GB if left unattended for longer than a week. It will retain these high levels of memory usage even while no operations are running. The only way to resolve the issue, is to restart the service, which currently happens on a Monday every week.

The environment is quite large with a Broker instance that manages over 1 million entities. However the server (both local and DB) has the specifications to deal with the load. Please see these below. It is also worth noting here also that to eliminate concurrency issues, the scheduling is setup to run everything sequentially (i.e UNIFYNow will step through each operation one at a time and no two operations will run at the same time both in MIM and UNIFYBroker).

Broker Server Specs:

CPU: 16 cores

Memory: 32GB

DB Server Specs:

CPU: 12 cores

Memory: 48GB

Although I haven't gathered this information for previous weeks, I have noticed some strange occurrences this morning and have documented them below:

  • Old LDAP Gateway connection that has not been closed:
  • Large number of SQL Connection to the UNIFYBroker DB:
  • High service memory usage while nothing is running (as mentioned before)

I will attach the logs and additional information below. Also in the logs below I have included information on the DB connections both before and after recycle the LDAP Gateway. But also note that recycling the LDAP Gateway also had no effect on the memory usage of the service. Let me know if there was anything else I can do to assist.

UNIFYBroker: v5.3.1

Affected Versions:
Fixed by Version:

Hi Hayden,

Initial analysis of the process dump revealed a large number of unreleased internal objects from the library Broker uses to execute PowerShell scripts, and seem to be related to remote PowerShell calls. Do any remote call get made in any of your PowerShell connectors/transformation/etc? If you could attach the scripts loaded by Broker that would also be helpful.

Can you check the default PowerShell version on that server? Run $PSVersionTable.PSVersion.ToString() in a PowerShell console, or (Get-Host).Version.ToString() if that happens to fails. I'll also need you to check that Broker is correctly using the same version. The easiest way of doing this is probably creating a new PowerShell connector with a field populated by the above command and running an import.

Next, can you check the versions of System.Management.Automation that Broker may be using. If you happen to have the .NET Framework SDK installed, you can use this command:

c:\Program Files (x86)\Microsoft SDKs\Windows\<version>\bin\NETFX <version> Tools\\gacutil.exe" -l System.Management.Automation

Since that's probably not the case, check the contents of this directory. It should exist with at least one versioned sub-directory containing a dll. Zipping the contents of this directory and attaching it here is probably the best option.

C:\windows\Microsoft.Net\assembly\GAC_MSIL\System.Management.Automation

Hi Beau,

Thanks for looking at that, that's interesting. As you have seen there are a large number of PowerShell scripts used in Broker so I will send all these over to you separately, many of which I haven't looked at before so there is a very real possibility these may be connections not being managed correctly within those scripts. But please find below the other details you asked for:

Default PowerShell version on server: 4.0

Broker Version: 4.0

System Automation dll:

v4.0_3.0.0.0__31bf3856ad364e35.zip

Let me know if there was anything else.

Thanks