Identity Broker Change Detection Failing Due to MS DTC
I'm having trouble overcoming the below issue which so far has appeared once or twice in the newly set up production system. I noticed this after being informed that a change made in CPAL had not been flown into FIM and other systems. Looking in the logs, there was one or two occurances of the below error.
20120703,02:37:01,Connector Processing started.,Connector Processor,Information,Connector Processing started for connector CPAL Person (page 1),Normal 20120703,02:37:02,Connector processing success.,Connector Processor,Information,"Processing page 1 for connector CPAL Person processed 277 entities, finding 2 differences. Duration: 00:00:01.2031712",Normal 20120703,02:37:02,Connector Post Processing started.,Connector Processor,Information,Connector Post Processing started for connector CPAL Person. Processed Entities: 227,Normal 20120703,02:37:02,Connector Post Processing success.,Connector Processor,Information,Connector Post Processing completed for connector CPAL Person. Processed Entities: 227. Matching Entities: 0. Reported Changes: 0. Duration: 00:00:00.1718816,Normal 20120703,02:37:02,Change detection engine import all items completed.,Change detection engine,Information,Change detection engine import all items for connector CPAL Person completed. Duration: 00:00:01.7500672,Normal 20120703,02:37:05,Started processing changes register items.,Change detection engine,Information,Started processing changes register items for connector CPAL Person.,Normal 20120703,04:07:05,Changes register item processing on failed.,Change detection engine,Warning,"Changes register item processing on connector CPAL Person failed with reason Communication with the underlying transaction manager has failed.. Duration: 00:00:00.0625024 Error details: System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed. ---> System.Runtime.InteropServices.COMException (0x8004D02B): The MSDTC transaction manager was unable to pull the transaction from the source transaction manager due to communication problems. Possible causes are: a firewall is present and it doesn't have an exception for the MSDTC process, the two machines cannot find each other by their NetBIOS names, or the support for network transactions is not enabled for one of the two transaction managers. (Exception from HRESULT: 0x8004D02B) at System.Transactions.Oletx.IDtcProxyShimFactory.ReceiveTransaction(UInt32 propgationTokenSize, Byte propgationToken, IntPtr managedIdentifier, Guid& transactionIdentifier, OletxTransactionIsolationLevel& isolationLevel, ITransactionShim& transactionShim) at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte propagationToken) --- End of inner exception stack trace --- at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte propagationToken) at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx) at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx) at System.Transactions.EnlistableStates.Promote(InternalTransaction tx) at System.Transactions.Transaction.Promote() at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction) at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte whereabouts) at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte whereAbouts) at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx) at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction) at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject) at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) at System.Data.SqlClient.SqlConnection.Open() at System.Data.SqlClient.SqlBulkCopy.CreateOrValidateConnection(String method) at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServer(Int32 columnCount) at System.Data.SqlClient.SqlBulkCopy.WriteToServer(DataTable table, DataRowState rowState) at Unify.Repository.ChangesItemContext.InsertItems(HashSet`1 addedItems, ChangesRegisterDataContext sourceContext, SqlConnection connection) at Unify.Data.LinqContextConversionBase`4.SubmitChanges() at Unify.Framework.BaseConnectorAdapterTransformationChangeProcessor.ProcessChangeReport(IDictionaryTwoPassDifferenceReport`4 changesReport, DateTime changeProcessTime) at Unify.Framework.ChangeReportProcessor.<>c__DisplayClassc.<ProcessCurrentReport>b__b(ITransformationChangeProcessor processor) at Unify.Framework.Visitor.<>c__DisplayClass1`1.<Visit>b__0(T item, Int32 index) at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor) at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`1 visitor) at Unify.Framework.ChangeReportProcessor.ProcessCurrentReport(IEnumerable`1 adapterTransformationProcessors, IDictionaryTwoPassDifferenceReport`4 differenceReport, DateTime changeTime) at Unify.Framework.ChangeReportProcessor.CreateAndProcessReport[T](IEnumerable`1 adapterTransformationProcessors, IEnumerable`1 sourceEnumerable, DateTime changeTime, Action`2 addAction) at Unify.Framework.ChangeReportProcessor.ProcessReport(IChangeReportProcessingRequest request) at Unify.Framework.Visitor.<>c__DisplayClass1`1.<Visit>b__0(T item, Int32 index) at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor) at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`1 visitor) at Unify.Framework.ChangeReportProcessor.RunBase() at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal
On the SQL Server System (a different virtual system), the below appears in the event log
Event 4879, MSDTC Client 2 MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system DSCSQL1.
DSCSQL1 is the database server and the system where the above error appeared.
MSDTC is enabled on both the SQL Server and the Identity Broker systems with "Network DTC Access", "Allow Inbound" and "Allow Outbound" are enabled. The same set up appeared in test.
I didn't see this issue in the test environment, I'm sure there is an environmental reason for it but I'm not sure what that is. After the error above Identity Broker appears to run along just fine, however updates may not have been made in the source system since the one above occurred.
Customer support service by UserEcho
On further investigation, if I do an adapter entity search in the management Studio, the record shows up. However Event Broker (3.0.1 Revision #3) hasn't picked up the changes.
Regarding the MSDTC setting too. Windows Firewall is disabled on both systems. On asking the SQL Admin, he wasnt aware of any other firewalls being in place.
Richard, did you have any luck following restarts of the machines?
Were you able to test using the DTCTester Tool?
It was too late to get a retest happening (prod systems owners werent in).
I came in this morning and have found the following error on one system
So that seems pretty clear about what should be tried next.
Adding and updating a user both functioned correctly after reinstalling MS DTC. Note I rebooted both after the uninstall, then again after the reinstall just to be sure. It was suggest at http://gaptheguru.wordpress.com/sql-server/msdtc-troubleshooting/ that reboots may be necessary.
That's good news, fantastic work.
When you are satisfied and are going to resolve, please let me know so that I can update relevant documentation.
I believe this is resolved. An additional delete was also detected. If you need any information Adam let me know. Based on the already posted error, it looked like a case of system cloning. The steps in the Windows Event Viewer worked (as noted, I rebooted after the uninstall then again after the reinstall of msdtc).
Note I dont think I saw that error message at this point yesterday. As such, it may be sporadic. We were getting MSDTC errors on both systems, but I only noticed this message on one of them.
From what I can tell it shouldnt matter what system you choose to reinstall ms dtc on...I did it only on the database server to limit impact to other services.