Bugs
Errors on LDAP gateway leave open LDAP active connections
Hi Team,
I have been noticing building active connections within UNIFYBroker (v5.3.2) for quite some time. This list continues to build endlessly, sometimes looking back and seeing > 20 active connections in an environment where all operations run sequentially. All LDAP connections are happening from MIM Sync.
I suspect there are 2 issue here, one that is generating the error in the first place, and the second where the connection is remaining open after a failure.
As you can see in the screen shot there is an active connection remaining open at 2:06 PM. This connection is then immediate follow by errors within the UNIFYBroker logs:
" A client has connected to the LDAP endpoint from address: 192.168.60.200:62993."
"An error occurred for gateway LDAP Gateway (1364c700-99c8-40aa-801d-0153427e62a9) on client from 192.168.60.200:62993. More details:
Unify.Product.IdentityBroker.PoorlyConstructedLDAPMessageException: The LDAP server for gateway LDAP Gateway (1364c700-99c8-40aa-801d-0153427e62a9) received a poorly constructed LDAP message and failed with the error: The LDAP message tag is unparsable.
at Unify.Product.IdentityBroker.LDAPConnection.ReadMessage()
at Unify.Product.IdentityBroker.LDAPConnection.TryReadMessage(RfcLdapMessage& message, RfcLdapResult& error)"
" Handling of LDAP extended request.
Handling of LDAP extended request from user Anonymous on connection 192.168.60.200:62993 failed with error "Authentication failed because the remote party has closed the transport stream.". Duration: 00:00:00.1018681."
"An error occurred for gateway LDAP Gateway (1364c700-99c8-40aa-801d-0153427e62a9) on client from 192.168.60.200:62993. More details:
Internal Server Error #11: System.Security.Authentication.AuthenticationException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest, Boolean renegotiation)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at Unify.Product.IdentityBroker.LDAPConnectionSecurityExtensions.TLSHandshake(ILDAPConnection connection, IRfcLdapMessage message, ISecurityEngine securityEngine)
at Unify.Product.IdentityBroker.StartTLSRequestHandlerSecurityDecorator.HandleRequest(IRfcLdapMessage message, CancellationToken token, Action`1 postAction)
at Unify.Product.IdentityBroker.LDAPRequestHandlerSecurityDecorator.HandleRequest(IRfcLdapMessage message, CancellationToken token, Action`1 postAction)
at Unify.Product.IdentityBroker.LDAPConnection.d__35.MoveNext()"
Customer support service by UserEcho
Morning Hayden,
Is there a particular MIM operation running that correlates to the error? From the stack trace it looks like something is trying to open an SSL session but not sending the correct payload or closing before the payload is received. Do you have a certificate configured for use in your LDAP gateway?
Hi Matt,
That particular failure I was able to see which job in particular triggered at the time. I rebuilt the schema/partitions on that MA, which was out of date, and so far the error hasn't reoccured for that MA. And yes there is a certificate configured.
What is also good to note is these errors don't always produce these residual connections. There are many occurrences in the logs where these errors are occuring and there are no active connections left open at that time.
Okay, that's good to hear.
UNIFYBroker does have a job which runs every minute, cleaning up connections that haven't had a message sent in the last 2 hours, or that have been closed by the client. It's possible that something is keeping that connection open for a period of time (or continuing to use the connection even though it's had an initial error) so that's why I was curious about the operation running for it.
I'll have a look at the MA to see if there's any scenarios under which it would keep a connection open. What's the oldest you've seen a connection be when looking at the list?
Thanks Matt, I don't have any examples of connections hanging around for longer than the above screenshot, but I think I've seen them hanging around for longer than a couple of days.