Questions
Associations Connector fails with "Only one use of each socket address is normally permitted"
We have just put a solution in UAT that was working correctly in Dev. The two Locations connectors (Person and Organization) have imported fine. The Associations connector runs for a couple of minutes then fails with the error below.
I have asked the CM admin to look for service API errors on their side, but also wanted to ask: does the Associations connector make multiple connections that could be over-lapping?
Change detection engine import all items failed. Change detection engine import all items for connector CM Associations failed with reason Unable to connect to the remote server. Duration: 00:01:38.0443655 Error details: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted 10.111.200.11:80 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.GetResponse() at ServiceStack.ServiceClientBase.Send[TResponse](String httpMethod, String relativeOrAbsoluteUrl, Object request) at Unify.Product.IdentityBroker.HPTrimV9WebCommunicator.<getalllocationuris>d__36.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext() at System.Linq.Buffer`1..ctor(IEnumerable`1 source) at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source) at Unify.Product.IdentityBroker.HPTrimV9WebCommunicator.InnerGetAssociations(IEnumerable`1 uri, IEnumerable`1 relationshipTypes, Int32 pageSize, IWebServiceCommunicatorInformation information, CancellationToken token) at Unify.Product.IdentityBroker.HPTrimV9WebCommunicator.GetAllAssociations(IEnumerable`1 relationshipTypes, Guid connectorId, String searchQuery, Int32 pageSize, IWebServiceCommunicatorInformation information, CancellationToken cancellationToken) at Unify.Product.IdentityBroker.HPTrimWebCommunicatorDecorator.GetAllAssociations(IEnumerable`1 relationshipTypes, Guid connectorId, String searchQuery, Int32 pageSize, IWebServiceCommunicatorInformation information, CancellationToken cancellationToken) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Unify.Product.IdentityBroker.HPTrimWebCommunicatorDecorator.GetAllAssociations(IEnumerable`1 relationshipTypes, Guid connectorId, String searchQuery, Int32 pageSize, IWebServiceCommunicatorInformation information, CancellationToken cancellationToken) at Unify.Product.IdentityBroker.HpTrimAgent.GetAllAssociations(IEnumerable`1 relationshipTypes, Guid connectorId, String searchQuery, Int32 pageSize, CancellationToken cancellationToken) at Unify.Product.IdentityBroker.HpTrimAssociationsConnector.GetAllEntities(IStoredValueCollection storedValueState, CancellationToken cancellationToken) at Unify.Product.IdentityBroker.ConnectorToReadingConnectorBridge.GetAllEntities(IStoredValueCollection storedValueState, CancellationToken cancellationToken) at Unify.Product.IdentityBroker.AuditReadingConnectorDecorator.GetAllEntities(IStoredValueCollection storedValues, CancellationToken cancellationToken) at Unify.Product.IdentityBroker.EventNotifierReadingConnectorDecoratorBase`1.GetAllEntities(IStoredValueCollection storedValues, CancellationToken cancellationToken) at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess() at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase() at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run() at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass30_0.<run>b__0() at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state) </run></getalllocationuris>
Answer
Based on the following web page I ran a NETSTAT and there's many, many, many TCP connections to the target with TIME_WAIT as a status. Something connected?
https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
Hi Carol,
The CM connector relies on the client library provided by HP so how it handles the connection is not something we are exposed to. We use the client correctly, however the process for getting associations is request heavy, though, so if the client does not handle its connections properly it could result in what you are seeing.
How many locations are you fetching associations for in UAT compared to DEV?
In Dev we started with an empty database and provisioned locations. In UAT there is a copy of the Prod database, so this is the level of data we will have in Prod.
There are 8000 Persons and 6600 Organizations. There will be other Location types too, I don't know how many because I only retrieve those two types. So in terms of the Associations connector I expect it be pulling back a lot more than 15,000 Association objects. I was only planning on updating Associations overnight so it only needs to run once a day.
I'm going to send an email to their CM consultants and see if there's any advice on their side about handling large queries. For instance making one or both of the changes suggested in the answer to this post: https://stackoverflow.com/questions/1339142/wcf-system-net-socketexception-only-one-usage-of-each-socket-address-protoco
Is there any association fields on the connector that are not needed? The connector needs to make a call per field, per location, so cutting the connector schema to only what's needed would greatly reducing the number of calls being made. This may be enough to be a temporary work around while I look into this issue further.
Removing un-needed schema did indeed allow the initial import to run, so thanks for that tip. I'll have to see what happens when I enable all connectors again. I do have the 3 CM connectors in an exclusion group.
Good to hear.
Looking into the client library, there's not much we can do about how it creates and manages the socket connections, unfortunately. I don't foresee any further problems since the import succeeded (other connector types would produce socket connections much less frequently, and the exclusion group would mitigate that in any case) so I'll make this as being answered.
I have created a backlog item to find a way to better mitigate or avoid the issue, so if any further issues arise let us know and we'll bump its priority.
Thanks, agree. I'll keep an eye on it and let you know if there are further issues. The method in the stackoverflow thread of increasing available sockets may be something I'll try if it comes to it.
A better option, also, would be if Micro Focus could fix their client library. Would it be possible to get the client to raise a bug request with MF?
I did send a query to their CM support mob, but they're another 3rd party like us, so probably just as hard as us trying to get MS code changed.
The first import ran but subsequent Full imports are failing (Deltas ok).
Is there any way to filter this connector? I'm getting over 25k Associations back. I'm only interested in associations for Person and Organization Locations.
The associations connector allow you to provide a search query to the api. The query format is documented in the api documentation, but this should be what you want for person and organization types:
type=Person or type=Organization
Customer support service by UserEcho
Is there any association fields on the connector that are not needed? The connector needs to make a call per field, per location, so cutting the connector schema to only what's needed would greatly reducing the number of calls being made. This may be enough to be a temporary work around while I look into this issue further.