Hi Gents,

I'm having an unusual issue with a custom connector. When running exports, a failure is occurring. However the single failure seems to be stopping subsequent entities from exporting. I currently have export 10 configured on the MA with a batch size of 1 (<-- which is interesting)

I've confirmed that the only results.SetFailed that is hit is the one in the UpdateEntity method. This is returned to the MA with an error of 'Other' and the actual exception is NOT included. After that point, no more entities are processed by the connector, and these show up as a 'cd-error' on the MA.

Nothing significant is noted in either the IDB or Event Logs.

Environment Details:

Running IDB v5.1.0 Revision #0

Patch: Unify.IdentityBroker.ChangeLog.Repository.Sql.dll

Unify.IdentityBroker.FIMAdapter

After a recent production incident where MIM kept presenting the same watermark to IDB (5.1) on delta imports there may be an opportunity for Identity Broker to handle the watermark storage in a better way that works around this MIM issue.

From talking the Curtis he mentioned that this issue has been seen with other clients, and the only workaround is to either re-create the MA or run a full import which in large environments may not be practical. Acknowledge that this is 100% a MIM issue, but could be a plus for the IDB if it can provide a workaround to such an issue that can have a big impact on large environments.

There are a few options that I could see:

- store the watermark in the MaData directory for the MA and use that instead

- store the watermark in the MaData directory for the MA and build some smarts around that watermark in combination with the MIM provided watermark.

It could be possible done by providing an option in the ECMA2 MA to enable/disable such enhanced functionality.

Using

• IdB v5.0.5 Revision #0
  Plugins
• Microsoft Active Directory 5.0.1.2
• Microsoft Azure AD Connector 5.0.1.5

The following was written this morning to the PROD IdB logs (verbose mode):

20161213,03:00:14,UNIFY Identity Broker,Service Engine,Warning,"An error occurred whilst coordinating the plug-in engine.  The error was:

System.BadImageFormatException: Could not load file or assembly 'file:///E:\Program Files\UNIFY Solutions\Identity Broker\Services\Unify.Service.Connect32.exe' or one of its dependencies. An attempt was made to load a program with an incorrect format.
File name: 'file:///E:\Program Files\UNIFY Solutions\Identity Broker\Services\Unify.Service.Connect32.exe'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)
   at System.Reflection.Assembly.LoadFrom(String assemblyFile)
   at Unify.Framework.ExtensibilityPlugInDictionary`4.InitializeAssemblyAttributeDefinition(String assemblyLocation)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
",Verbose

Multiple occurrences of this issue in the 24 hour period just completed (log file just rolled over).

See corresponding JIRA issue QBE-65 for log file attachment.

Over the last couple of weeks troubleshooting a separate issue, the customer has been reporting that some users hadn't been assigned licenses. In some cases this was due to accounts in error, but in others it has turned out to be that the delta imports from the corresponding IdentityBroker adapter (run on change detection by Event Broker) have not included changes that they should have. Such changes are only surfaced to MIM after a MIM MA full import (delta sync) is subsequently run.

To mitigate this problem, a twice-daily full import/delta sync operation across both IdB adapters is being performed. Over the last 4 days since this has been in place, a number of changes continue to be surfaced in the full import some 10 minutes after the last delta import. The latest of these FI runs which returned 5 changes was run at 7:07 am on Saturday 3rd December - at a time when there should be no business activity happening in any geographic region for QBE.

Investigations into SQL queries such as the following identified identities where a MODIFY change entry was present without a corresponding INSERT:

SELECT *  FROM [Unify.IdentityBroker50].[dbo].[ChangeLog]
WHERE TargetDistinguishedName = 'CN=bob,OU=container,DC=IdentityBroker' 

Filing this against the O365 connector because it happened to be for the related adapter - but it is likely this is a generic problem unrelated to a specific connector (i.e. am seeing FI steps return records consistently for both AD and AAD-based adapters)

Re-opened issue QBE-51 previously raised for this issue.

When viewing the VERBOSE logs today via the console, and clicking NEXT, the following exception was raised:

The logs are also attached from the time of the error here: Log file archive

QBE reported this week that they are continuing to have long periods (several hours) where licenses are not being assigned, and are having to manually restart the IdB service multiple times during the day (the service is already being restarted each night at 4 am). The timeout error continues to be reported in the logs

See JIRA ticket QBE-59 for more details.

QBE reported an ma-extension-error export failure for 702 O365 license updates this morning, but on later inspection the errors appeared to resolve themselves. However on further inspection the Application Event Log revealed corresponding Index was out of range exception within the Identity Broker for Office Enterprise 5.0.1.5 connector logic.

Refer to QBE JIRA ticket QBE-64

Receive an Export error "ma-extension-error" for the Global Expert Person MA when user "UID=53307" is to be created in Expertv8 via UNIFY Identity Broker for Expertv8 4.1.0.0.

The error details in FIM is as follow:

System.Exception: Error occurred when attempting to save entity with distinguished name

UID=53307

Error:

String or binary data would be truncated.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at System.Data.Linq.SqlClient.SqlProvider.Execute(Expression query, QueryInfo queryInfo, IObjectReaderFactory factory, Object[] parentArgs, Object[] userArgs, ICompiledSubQuery[] subQueries, Object lastResult)
   at System.Data.Linq.SqlClient.SqlProvider.ExecuteAll(Expression query, QueryInfo[] queryInfos, IObjectReaderFactory factory, Object[] userArguments, ICompiledSubQuery[] subQueries)
   at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
   at System.Data.Linq.ChangeDirector.StandardChangeDirector.DynamicInsert(TrackedObject item)
   at System.Data.Linq.ChangeDirector.StandardChangeDirector.Insert(TrackedObject item)
   at System.Data.Linq.ChangeProcessor.SubmitChanges(ConflictMode failureMode)
   at System.Data.Linq.DataContext.SubmitChanges(ConflictMode failureMode)
   at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.SaveEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.EventNotifierAddingConnectorDecorator.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& backwardAdapterEntities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportAdd(IAdapter adapter, IAdapterEntitySaveChange pendingAdd)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)
   at SyncInvokeExportChanges(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
   at Unify.Product.IdentityBroker.IdentityBrokerManagementAgentProxy.ExportEntry(ModificationType modificationType, String[] changedAttributes, CSEntry csentry)

Error in the IdB log file:

20161101,00:04:30,UNIFY Identity Broker,Connector,Warning,"Add entities to connector failed.
Add entities [Count:1] to connector Global Aderant Expert Connector failed with reason String or binary data would be truncated.
The statement has been terminated.. Duration: 00:00:01.6170000
Error details:
System.Data.SqlClient.SqlException (0x80131904): String or binary data would be truncated.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at System.Data.Linq.SqlClient.SqlProvider.Execute(Expression query, QueryInfo queryInfo, IObjectReaderFactory factory, Object[] parentArgs, Object[] userArgs, ICompiledSubQuery[] subQueries, Object lastResult)
   at System.Data.Linq.SqlClient.SqlProvider.ExecuteAll(Expression query, QueryInfo[] queryInfos, IObjectReaderFactory factory, Object[] userArguments, ICompiledSubQuery[] subQueries)
   at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
   at System.Data.Linq.ChangeDirector.StandardChangeDirector.DynamicInsert(TrackedObject item)
   at System.Data.Linq.ChangeDirector.StandardChangeDirector.Insert(TrackedObject item)
   at System.Data.Linq.ChangeProcessor.SubmitChanges(ConflictMode failureMode)
   at System.Data.Linq.DataContext.SubmitChanges(ConflictMode failureMode)
   at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.SaveEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.EventNotifierAddingConnectorDecorator.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& backwardAdapterEntities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportAdd(IAdapter adapter, IAdapterEntitySaveChange pendingAdd)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)
   at SyncInvokeExportChanges(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)",Normal
20161101,00:04:30,UNIFY Identity Broker,Adapter,Warning,"Adapter request to add entity to adapter space failed.
Adapter request to add entity c47170eb-9ad8-4289-ba52-081a4fef9154 to adapter Global Aderant Expert Employee (22ba5a36-1316-45c9-9926-dc5307de86c0) failed with reason String or binary data would be truncated.
The statement has been terminated.. Duration: 00:00:01.8620000
Error details:
System.Data.SqlClient.SqlException (0x80131904): String or binary data would be truncated.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at System.Data.Linq.SqlClient.SqlProvider.Execute(Expression query, QueryInfo queryInfo, IObjectReaderFactory factory, Object[] parentArgs, Object[] userArgs, ICompiledSubQuery[] subQueries, Object lastResult)
   at System.Data.Linq.SqlClient.SqlProvider.ExecuteAll(Expression query, QueryInfo[] queryInfos, IObjectReaderFactory factory, Object[] userArguments, ICompiledSubQuery[] subQueries)
   at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
   at System.Data.Linq.ChangeDirector.StandardChangeDirector.DynamicInsert(TrackedObject item)
   at System.Data.Linq.ChangeDirector.StandardChangeDirector.Insert(TrackedObject item)
   at System.Data.Linq.ChangeProcessor.SubmitChanges(ConflictMode failureMode)
   at System.Data.Linq.DataContext.SubmitChanges(ConflictMode failureMode)
   at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.SaveEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.EventNotifierAddingConnectorDecorator.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& backwardAdapterEntities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportAdd(IAdapter adapter, IAdapterEntitySaveChange pendingAdd)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)
   at SyncInvokeExportChanges(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)",Normal
20161101,00:05:39,UNIFY Identity Broker,Logging Engine,Information,Log file started.,Minimal

and the error is as follow:

<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>131076</EventID>
<Type>3</Type>
<SubType Name="Error">0</SubType>
<Level>2</Level>
<TimeCreated SystemTime="2016-11-01T00:04:30.2193430Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{1025fb0d-a6b4-4b7a-9c37-25aa7755cdec}" />
<Execution ProcessName="Unify.Service.Connect" ProcessID="12704" ThreadID="64" />
<Channel />
<Computer>TSTDC1FIMWEB02</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/en-AU/library/System.ServiceModel.Diagnostics.TraceHandledException.aspx</TraceIdentifier>
<Description>Handling an exception.</Description>
<AppDomain>Unify.Service.Connect.exe</AppDomain>
<Exception>
<ExceptionType>System.Data.SqlClient.SqlException, System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>String or binary data would be truncated.
The statement has been terminated.</Message>
<StackTrace>
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at System.Data.Linq.SqlClient.SqlProvider.Execute(Expression query, QueryInfo queryInfo, IObjectReaderFactory factory, Object[] parentArgs, Object[] userArgs, ICompiledSubQuery[] subQueries, Object lastResult)
at System.Data.Linq.SqlClient.SqlProvider.ExecuteAll(Expression query, QueryInfo[] queryInfos, IObjectReaderFactory factory, Object[] userArguments, ICompiledSubQuery[] subQueries)
at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
at System.Data.Linq.ChangeDirector.StandardChangeDirector.DynamicInsert(TrackedObject item)
at System.Data.Linq.ChangeDirector.StandardChangeDirector.Insert(TrackedObject item)
at System.Data.Linq.ChangeProcessor.SubmitChanges(ConflictMode failureMode)
at System.Data.Linq.DataContext.SubmitChanges(ConflictMode failureMode)
at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.SaveEntities(IEnumerable`1 entities)
at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.AddEntities(IEnumerable`1 entities)
at Unify.Product.IdentityBroker.EventNotifierAddingConnectorDecorator.AddEntities(IEnumerable`1 entities)
at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& backwardAdapterEntities)
at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities)
at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity)
at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportAdd(IAdapter adapter, IAdapterEntitySaveChange pendingAdd)
at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)
at SyncInvokeExportChanges(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
</StackTrace>
<ExceptionString>System.Data.SqlClient.SqlException (0x80131904): String or binary data would be truncated.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at System.Data.Linq.SqlClient.SqlProvider.Execute(Expression query, QueryInfo queryInfo, IObjectReaderFactory factory, Object[] parentArgs, Object[] userArgs, ICompiledSubQuery[] subQueries, Object lastResult)
   at System.Data.Linq.SqlClient.SqlProvider.ExecuteAll(Expression query, QueryInfo[] queryInfos, IObjectReaderFactory factory, Object[] userArguments, ICompiledSubQuery[] subQueries)
   at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
   at System.Data.Linq.ChangeDirector.StandardChangeDirector.DynamicInsert(TrackedObject item)
   at System.Data.Linq.ChangeDirector.StandardChangeDirector.Insert(TrackedObject item)
   at System.Data.Linq.ChangeProcessor.SubmitChanges(ConflictMode failureMode)
   at System.Data.Linq.DataContext.SubmitChanges(ConflictMode failureMode)
   at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.SaveEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AderantExpert8EmployeeConnector.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.EventNotifierAddingConnectorDecorator.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& backwardAdapterEntities)
   at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportAdd(IAdapter adapter, IAdapterEntitySaveChange pendingAdd)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)
   at SyncInvokeExportChanges(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)</ExceptionString>
<DataItems>
<Data>
<Key>HelpLink.ProdName</Key>
<Value>Microsoft SQL Server</Value>
</Data>
<Data>
<Key>HelpLink.ProdVer</Key>
<Value>12.00.4100</Value>
</Data>
<Data>
<Key>HelpLink.EvtSrc</Key>
<Value>MSSQLServer</Value>
</Data>
<Data>
<Key>HelpLink.EvtID</Key>
<Value>8152</Value>
</Data>
<Data>
<Key>HelpLink.BaseHelpUrl</Key>
<Value>http://go.microsoft.com/fwlink</Value>
</Data>
<Data>
<Key>HelpLink.LinkId</Key>
<Value>20476</Value>
</Data>
</DataItems>
</Exception>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
</E2ETraceEvent>

It does not look like the Check operation is working, I know there are changes currently pending but Event broker does not think so.

Log below from Event Broker

20161031,05:54:34,UNIFY FIM Event Broker,Operation List Executor,Information,Operation list FIM Agent Callista Delta Import Sync started,Verbose
20161031,05:54:34,UNIFY FIM Event Broker,Operation List Executor,Information,Running check operation Identity Broker Changes for operation list FIM Agent Callista Delta Import Sync,Verbose
20161031,05:54:34,UNIFY FIM Event Broker,Operation List Executor,Information,Check operation Identity Broker Changes for operation list FIM Agent Callista Delta Import Sync returned False,Verbose
20161031,05:54:34,UNIFY FIM Event Broker,Operation List Executor,Information,Operation list FIM Agent Callista Delta Import Sync finished,Verbose

@Northcott, one fine day, IdB started throwing these errors on all its connectors.

Error in One liner:

Change detection engine import all items failed.
Change detection engine import all items for connector SecurityUser failed with reason Aurion API error -1: . Duration: 00:00:01.0940697

And to Note: This error is happening in all the Connectors(Schedule, Person, Sec User and Position)

Work did from my end:

1. Tested Connectivity from Aurion Agent and it is connecting successfully

2. Tested the URL and I am able to see the XML output.

3. Tried restarting the IdB Service and still the error pops up

Some Investigation from my side, This error started with HTTP404 error which was thrown on Schedule Connector

Import all entities from connector failed.Import all entities from connector Schedule failed with reason The request failed with HTTP status 404: Not Found.. Duration: 00:00:00.0270257
Error details:
System.Net.WebException: The request failed with HTTP status 404: Not Found.
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at Unify.Communicators.AurionAPI.EV397_AURION_WSService.LOGON(String P_USER, String P_PASSWORD, String& P_TOKEN, String& P_MESSAGE, Decimal& P_STATUS)
at Unify.Communicators.AurionWSCommunicator.Logon(String userName, String password)
at Unify.Communicators.AurionAgent.Open()
at Unify.Connectors.AurionApiReadingConnector.<GetAllEntities>d__4.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Unify.Framework.Collections.ActionOnExceptionEnumerator`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ActionOnLast>d__10`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ProduceAutoPages>d__7`1.MoveNext()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPool[T](IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass33_0.<Run>b__0()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)

And ever since that happened, the below error started to happen.

Change detection engine import all items failed.Change detection engine import all items for connector SecurityUser failed with reason Aurion API error -1: . Duration: 00:00:01.0940697
Error details:
System.Exception: Aurion API error -1: 
at Unify.Communicators.AurionWSCommunicator.RaiseExceptionOnErrorCondition(Decimal pStatus, String pMessage)
at Unify.Communicators.AurionWSCommunicator.SetFunction(String function)
at Unify.Communicators.AurionApiFunctionBase.RunFunction()
at Unify.Communicators.AurionAgent.QueryToXml(String queryId, String expectedObjectName)
at Unify.Connectors.AurionApiReadingConnector.<GetAllEntities>d__4.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Unify.Framework.Collections.ActionOnExceptionEnumerator`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ActionOnLast>d__10`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ProduceAutoPages>d__7`1.MoveNext()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPool[T](IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c__DisplayClass33_0.<Run>b__0()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)

P.S. Please let me know if you need the configuration and logs. I cannot find an attachment where I can attach the logs