I have been tracking this issue for a few weeks within an environment. And what I have been noticing is the performance of the UNIFYBroker service deteriorating over time. While there is an existing work around, it would be good to find the root cause of this, whether it be a poorly configured item somewhere or an underlying issue.
Currently in the environment we notice the service become slower, at a minimum, over the course of about a week. It will gradually take sets of operation lists longer and longer to run. More specifically this will be worse on Monday after all the Full Baseline operations run over the weekend. And then gradually get even worse from there. The only correlation I can see so far is during the times it is running slow, is the service will be using upward of 5GB of ram, even getting to over 10GB if left unattended for longer than a week. It will retain these high levels of memory usage even while no operations are running. The only way to resolve the issue, is to restart the service, which currently happens on a Monday every week.
The environment is quite large with a Broker instance that manages over 1 million entities. However the server (both local and DB) has the specifications to deal with the load. Please see these below. It is also worth noting here also that to eliminate concurrency issues, the scheduling is setup to run everything sequentially (i.e UNIFYNow will step through each operation one at a time and no two operations will run at the same time both in MIM and UNIFYBroker).
Broker Server Specs:
CPU: 16 cores
DB Server Specs:
CPU: 12 cores
Although I haven't gathered this information for previous weeks, I have noticed some strange occurrences this morning and have documented them below:
- Old LDAP Gateway connection that has not been closed:
- Large number of SQL Connection to the UNIFYBroker DB:
- High service memory usage while nothing is running (as mentioned before)
I will attach the logs and additional information below. Also in the logs below I have included information on the DB connections both before and after recycle the LDAP Gateway. But also note that recycling the LDAP Gateway also had no effect on the memory usage of the service. Let me know if there was anything else I can do to assist.
Customer support service by UserEcho