1. UNIFYBroker Forum
  2. Questions
0
Answered

Identity Broker Change Detection Failing Due to MS DTC

Richard Courtenay 12 years ago • updated by anonymous 9 years ago • 7

I'm having trouble overcoming the below issue which so far has appeared once or twice in the newly set up production system. I noticed this after being informed that a change made in CPAL had not been flown into FIM and other systems. Looking in the logs, there was one or two occurances of the below error.

20120703,02:37:01,Connector Processing started.,Connector Processor,Information,Connector Processing started for connector CPAL Person (page 1),Normal
20120703,02:37:02,Connector processing success.,Connector Processor,Information,"Processing page 1 for connector CPAL Person processed 277 entities, finding 2 differences. Duration: 00:00:01.2031712",Normal
20120703,02:37:02,Connector Post Processing started.,Connector Processor,Information,Connector Post Processing started for connector CPAL Person. Processed Entities: 227,Normal
20120703,02:37:02,Connector Post Processing success.,Connector Processor,Information,Connector Post Processing completed for connector CPAL Person. Processed Entities: 227. Matching Entities: 0. Reported Changes: 0. Duration: 00:00:00.1718816,Normal
20120703,02:37:02,Change detection engine import all items completed.,Change detection engine,Information,Change detection engine import all items for connector CPAL Person completed. Duration: 00:00:01.7500672,Normal
20120703,02:37:05,Started processing changes register items.,Change detection engine,Information,Started processing changes register items for connector CPAL Person.,Normal
20120703,04:07:05,Changes register item processing on failed.,Change detection engine,Warning,"Changes register item processing on connector CPAL Person failed with reason Communication with the underlying transaction manager has failed.. Duration: 00:00:00.0625024
Error details:
System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed. ---> System.Runtime.InteropServices.COMException (0x8004D02B): The MSDTC transaction manager was unable to pull the transaction from the source transaction manager due to communication problems. Possible causes are: a firewall is present and it doesn't have an exception for the MSDTC process, the two machines cannot find each other by their NetBIOS names, or the support for network transactions is not enabled for one of the two transaction managers. (Exception from HRESULT: 0x8004D02B)
   at System.Transactions.Oletx.IDtcProxyShimFactory.ReceiveTransaction(UInt32 propgationTokenSize, Byte[] propgationToken, IntPtr managedIdentifier, Guid& transactionIdentifier, OletxTransactionIsolationLevel& isolationLevel, ITransactionShim& transactionShim)
   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)
   --- End of inner exception stack trace ---
   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)
   at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)
   at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)
   at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)
   at System.Transactions.Transaction.Promote()
   at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)
   at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)
   at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte[] whereAbouts)
   at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
   at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)
   at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
   at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
   at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
   at System.Data.SqlClient.SqlConnection.Open()
   at System.Data.SqlClient.SqlBulkCopy.CreateOrValidateConnection(String method)
   at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServer(Int32 columnCount)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServer(DataTable table, DataRowState rowState)
   at Unify.Repository.ChangesItemContext.InsertItems(HashSet`1 addedItems, ChangesRegisterDataContext sourceContext, SqlConnection connection)
   at Unify.Data.LinqContextConversionBase`4.SubmitChanges()
   at Unify.Framework.BaseConnectorAdapterTransformationChangeProcessor.ProcessChangeReport(IDictionaryTwoPassDifferenceReport`4 changesReport, DateTime changeProcessTime)
   at Unify.Framework.ChangeReportProcessor.<>c__DisplayClassc.<ProcessCurrentReport>b__b(ITransformationChangeProcessor processor)
   at Unify.Framework.Visitor.<>c__DisplayClass1`1.<Visit>b__0(T item, Int32 index)
   at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
   at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`1 visitor)
   at Unify.Framework.ChangeReportProcessor.ProcessCurrentReport(IEnumerable`1 adapterTransformationProcessors, IDictionaryTwoPassDifferenceReport`4 differenceReport, DateTime changeTime)
   at Unify.Framework.ChangeReportProcessor.CreateAndProcessReport[T](IEnumerable`1 adapterTransformationProcessors, IEnumerable`1 sourceEnumerable, DateTime changeTime, Action`2 addAction)
   at Unify.Framework.ChangeReportProcessor.ProcessReport(IChangeReportProcessingRequest request)
   at Unify.Framework.Visitor.<>c__DisplayClass1`1.<Visit>b__0(T item, Int32 index)
   at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
   at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`1 visitor)
   at Unify.Framework.ChangeReportProcessor.RunBase()
   at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal

On the SQL Server System (a different virtual system), the below appears in the event log

Event 4879, MSDTC Client 2
MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system DSCSQL1.

DSCSQL1 is the database server and the system where the above error appeared.

MSDTC is enabled on both the SQL Server and the Identity Broker systems with "Network DTC Access", "Allow Inbound" and "Allow Outbound" are enabled. The same set up appeared in test.

I didn't see this issue in the test environment, I'm sure there is an environmental reason for it but I'm not sure what that is. After the error above Identity Broker appears to run along just fine, however updates may not have been made in the source system since the one above occurred.

Richard Courtenay 12 years ago

On further investigation, if I do an adapter entity search in the management Studio, the record shows up. However Event Broker (3.0.1 Revision #3) hasn't picked up the changes.

Richard Courtenay 12 years ago

Regarding the MSDTC setting too. Windows Firewall is disabled on both systems. On asking the SQL Admin, he wasnt aware of any other firewalls being in place.

Adam van Vliet 12 years ago

Richard, did you have any luck following restarts of the machines?

Were you able to test using the DTCTester Tool?

Richard Courtenay 12 years ago

It was too late to get a retest happening (prod systems owners werent in).

I came in this morning and have found the following error on one system

The local MS DTC detected that the MS DTC on DSCSQL1 has the same unique identity as the local MS DTC. This means that the two MS DTC will not be able to communicate with each other. This problem typically occurs if one of the systems were cloned using unsupported cloning tools. MS DTC requires that the systems be cloned using supported cloning tools such as SYSPREP. Running 'msdtc -uninstall' and then 'msdtc -install' from the command prompt will fix the problem. Note: Running 'msdtc -uninstall' will result in the system losing all MS DTC configuration information.

So that seems pretty clear about what should be tried next.

Richard Courtenay 12 years ago

Adding and updating a user both functioned correctly after reinstalling MS DTC. Note I rebooted both after the uninstall, then again after the reinstall just to be sure. It was suggest at http://gaptheguru.wordpress.com/sql-server/msdtc-troubleshooting/ that reboots may be necessary.

Adam van Vliet 12 years ago

That's good news, fantastic work.

When you are satisfied and are going to resolve, please let me know so that I can update relevant documentation.

Richard Courtenay 12 years ago

I believe this is resolved. An additional delete was also detected. If you need any information Adam let me know. Based on the already posted error, it looked like a case of system cloning. The steps in the Windows Event Viewer worked (as noted, I rebooted after the uninstall then again after the reinstall of msdtc).

Note I dont think I saw that error message at this point yesterday. As such, it may be sporadic. We were getting MSDTC errors on both systems, but I only noticed this message on one of them.

From what I can tell it shouldnt matter what system you choose to reinstall ms dtc on...I did it only on the database server to limit impact to other services.

Customer support service by UserEcho