Dynamics CRM Metadata contains a reference that cannot be resolved SSL Problem

Using the Broker Microsoft Dynamics CRM v5.2.0.1, Infrastructure are seeing some intermittent errors. The error shows in MIM and when checking the IdB logs the content of the error is the same as what shows in IdB. It's not that big a problem. It only occurs occurs on export and the pending export that fails remains a pending export and is processed ten minutes later and the error isn't rethrown on the second export. It seems to be some type of network connection problem but there aren't a lot of settings to configure it in the CRM Agent. Just the address and account and they're both correct. The full error is pasted below.

System.InvalidOperationException: Metadata contains a reference that cannot be resolved: 'https://dynamicscrm.internal.dotars.gov.au/DAMS//XRMServices/2011/Organization.svc?wsdl&sdkversion=8.2'. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Description.MetadataExchangeClient.MetadataLocationRetriever.DownloadMetadata(TimeoutHelper timeoutHelper)
at System.ServiceModel.Description.MetadataExchangeClient.MetadataRetriever.Retrieve(TimeoutHelper timeoutHelper)
--- End of inner exception stack trace ---
at System.ServiceModel.Description.MetadataExchangeClient.MetadataRetriever.Retrieve(TimeoutHelper timeoutHelper)
at System.ServiceModel.Description.MetadataExchangeClient.ResolveNext(ResolveCallState resolveCallState)
at System.ServiceModel.Description.MetadataExchangeClient.GetMetadata(MetadataRetriever retriever)
at Microsoft.Xrm.Sdk.Client.ServiceMetadataUtility.RetrieveServiceEndpointMetadata(Type contractType, Uri serviceUri, Boolean checkForSecondary)
at Microsoft.Xrm.Sdk.Client.ServiceConfiguration`1..ctor(Uri serviceUri, Boolean checkForSecondary)
at Microsoft.Xrm.Sdk.Client.OrganizationServiceConfiguration..ctor(Uri serviceUri, Boolean enableProxyTypes, Assembly assembly)
at Microsoft.Xrm.Sdk.Client.ServiceConfigurationFactory.CreateConfiguration[TService](Uri serviceUri, Boolean enableProxyTypes, Assembly assembly)
at Unify.Product.IdentityBroker.OrganizationServiceCommunicator.GetOrganizationService(IAddressCommunicatorInformation communicatorInformation)
at Unify.Product.IdentityBroker.OrganizationServiceCommunicator.<>c__DisplayClass1_0.<.ctor>b__0()
at Unify.Product.IdentityBroker.AddressCommunicatorBase`2.get_Service()
at Unify.Product.IdentityBroker.DynamicsCrmAgent.GetAttributeMetadata(String objectName, EntityFilters schemaRetrieveEntityFilters)
at Unify.Product.IdentityBroker.DynamicsCrmAgent.RetrieveSpecialFieldTypes(String objectName)
at Unify.Product.IdentityBroker.DynamicsCrmObjectConnector.GetSpecialFieldTypesInformation(IDynamicsCrmAgent`2 agent)
at System.Lazy`1.CreateValue()
at System.Lazy`1.LazyInitValue()
at Unify.Product.IdentityBroker.DynamicsCrmObjectConnector.UpdateEntities(IEnumerable`1 entities, IEnumerable`1 originalEntities, ISaveEntityResults`2 results)
at Unify.Product.IdentityBroker.AuditUpdatingConnectorDecorator.UpdateEntities(IEnumerable`1 entities, IEnumerable`1 originalEntities, ISaveEntityResults`2 results)
at Unify.Product.IdentityBroker.EventNotifierUpdatingConnectorDecorator.UpdateEntities(IEnumerable`1 entities, IEnumerable`1 originalEntities, ISaveEntityResults`2 results)

Any clues on the resolution of this intermittent issue? We haven't done diagnostic logging because it's in production and the error is intermittent so making huge logs is undesirable. I googled a bit and it looks like there are lines of code to set the TLS version to 1.2 which has resolved the same error in different contexts for other people. But you guys don't hardcode the authentication with web services right? So maybe the bindings should be updated? Still it doesn't make much sense that it fails sometimes and works sometimes. Makes me think the service is the problem rather than broker.