Monash needs a transformation in adaptor to join two tables and group the resolved values (type of String) to a multi-value string attribute.

I looked at GROUP transformation but it only groups values to multi-value DN.

Please suggest the alternative if any or request additional transformation

The XML Blob for organization in objectclass user is not working as expected. Having exported-changes-not-imported issues

Expo

Hi

Looking for a migration guide on how to migrate IdB Settings / MA etc across environments.

Regards

We use the SQL agent in IdB to read and write from a simulated SAP data source for testing. When exporting changes to the database via IdB, after approximately 100 exports, IdB starts reporting the following exception. When the IdB is stopped and restarted, a further 100 can be exported before the error reappears.

System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)   at System.Data.SqlClient.SqlConnection.Open()   at Unify.Framework.Data.DbConnectionExtensions.OpenIfClosed(IDbConnection connection)   at Unify.Framework.Data.SqlServerDataControl.CreateComponent(Boolean factoryInformation)   at Unify.Framework.Data.AdoNetDataControl.ExecuteDataSetQuery(DataSet resultSet, String commandText, IEnumerable`1 parameters, String[] tableNames, CommandType commandType)   at Unify.Framework.Data.AdoNetDataControl.ReturnDataSetQueryT(String commandText, IEnumerable`1 parameters, String[] tableNames, CommandType commandType)   at Unify.Product.IdentityBroker.DirectCommunicatorBase`1.GetEntitiesDataTable(ICollection`1 keyList)   at Unify.Product.IdentityBroker.DirectCommunicatorBase`1.UpdateEntities(IEnumerable`1 entities)   at Unify.Product.IdentityBroker.DirectReadWriteConnectorBase`1.UpdateEntities(IEnumerable`1 entities)   at Unify.Product.IdentityBroker.ConnectorToUpdatingConnectorBridge.UpdateEntities(IEnumerable`1 entities)   at Unify.Product.IdentityBroker.EventNotifierUpdatingConnectorDecorator.UpdateEntities(IEnumerable`1 entities)   at Unify.Product.IdentityBroker.Adapter.UpdateEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& connectorEntities)   at Unify.Product.IdentityBroker.Adapter.UpdateEntities(IEnumerable`1 entities)   at Unify.Product.IdentityBroker.Adapter.UpdateEntity(IAdapterEntity entity)   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.UpdateEntity(IAdapterEntity entityToSave)   at Unify.Product.IdentityBroker.CompositeAdapter.UpdateEntity(IAdapterEntity entity)   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.UpdateEntity(IAdapterEntity entityToSave)   at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportUpdate(IAdapter adapter, IAdapterEntitySaveChange pendingUpdate)   at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)   at Unify.Product.IdentityBroker.LDIFAdapterServiceHostDecoratorBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)   at SyncInvokeExportChanges(Object , Object[] , Object[] )   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)

Request to import all entities from connector gappsCalendarConnector.",Normal
20150618,04:51:49,UNIFY Identity Broker,Connector,Warning,"Import all entities from connector failed.
Import all entities from connector gappsCalendarConnector failed with reason A Google Data exception was thrown for call CalendarResources.RetrieveAllCalendarResources with message ""Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/"". See inner exception for details. Processing continued: False. Response: Version 3.0 is not supported.. Duration: 00:00:07.2319590
Error details:
System.Exception: A Google Data exception was thrown for call CalendarResources.RetrieveAllCalendarResources with message ""Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/"". See inner exception for details. Processing continued: False. Response: Version 3.0 is not supported. ---> Google.GData.Client.GDataRequestException: Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/ ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Google.GData.Client.GDataRequest.Execute()
— End of inner exception stack trace —
at Google.GData.Apps.AppsPropertyService.QueryExtendedFeed(Uri uri, Boolean shouldGetAllPages)
at Google.GData.Apps.CalendarResourceService.RetrieveAllCalendarResources()
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetryTResult(String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
— End of inner exception stack trace —
at Unify.Product.IdentityBroker.GoogleAgent.<>c_DisplayClassab`1.<BackoffRetry>b_a5()
at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetryTResult(String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
at Unify.Product.IdentityBroker.GoogleAgent.<CalendarGetPages>d__23.MoveNext()
at Unify.Product.IdentityBroker.GoogleCalendarConnector.<GetAllEntities>d__15.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Unify.Framework.Collections.ActionOnExceptionEnumerator`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ActionOnLast>d__19`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ProduceAutoPages>d__a`1.MoveNext()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPoolT(IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Framework.Visitor.VisitEvaluateOnThreadPoolT(IEnumerable`1 visitCollection, Action`1 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.JobBase.Run()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c_DisplayClass29.<Run>b_27()
at Unify.Framework.ActionJob.RunBase()
at Unify.Framework.JobBase.Run()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal
20150618,04:51:49,UNIFY Identity Broker,Change detection engine,Error,"Change detection engine import all items failed.
Change detection engine import all items for connector gappsCalendarConnector failed with reason A Google Data exception was thrown for call CalendarResources.RetrieveAllCalendarResources with message ""Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/"". See inner exception for details. Processing continued: False. Response: Version 3.0 is not supported.. Duration: 00:00:07.2739485
Error details:
System.Exception: A Google Data exception was thrown for call CalendarResources.RetrieveAllCalendarResources with message ""Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/"". See inner exception for details. Processing continued: False. Response: Version 3.0 is not supported. ---> Google.GData.Client.GDataRequestException: Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/ ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Google.GData.Client.GDataRequest.Execute()
— End of inner exception stack trace —
at Google.GData.Apps.AppsPropertyService.QueryExtendedFeed(Uri uri, Boolean shouldGetAllPages)
at Google.GData.Apps.CalendarResourceService.RetrieveAllCalendarResources()
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetryTResult(String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
— End of inner exception stack trace —
at Unify.Product.IdentityBroker.GoogleAgent.<>c_DisplayClassab`1.<BackoffRetry>b_a5()
at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetryTResult(String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
at Unify.Product.IdentityBroker.GoogleAgent.<CalendarGetPages>d__23.MoveNext()
at Unify.Product.IdentityBroker.GoogleCalendarConnector.<GetAllEntities>d__15.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Unify.Framework.Collections.ActionOnExceptionEnumerator`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ActionOnLast>d__19`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ProduceAutoPages>d__a`1.MoveNext()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPoolT(IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Framework.Visitor.VisitEvaluateOnThreadPoolT(IEnumerable`1 visitCollection, Action`1 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.JobBase.Run()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c_DisplayClass29.<Run>b_27()
at Unify.Framework.ActionJob.RunBase()
at Unify.Framework.JobBase.Run()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal

We are seeing that googleapps MA is reporting errors in FIM in async mode.

Wanted to get more insight on why this is happening because as per our understanding there should be no error reporting in async mode.

Attached Unify Log from yesterday with loads of errors..

Attaching screenshot of errors in FIM

The error from one of them (reported in FIM)

System.Exception: Error occurred when attempting to save entity with distinguished name

CN=75fd882f-a487-4e03-9a77-5d3af811a8e1,OU=users

Error:

A Google API exception was thrown for call Users.Update with message "Google.Apis.Requests.RequestError
Entity already exists. [409]
Errors [
	Message[Entity already exists.] Location[ - ] Reason[duplicate] Domain[global]
]
". See inner exception for details. Processing continued: False.
   at Unify.Product.IdentityBroker.GoogleAgent.<>c__DisplayClassb1`1.<BackoffRetry>b__a9()
   at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)
   at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetry[TResult](String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
   at Unify.Product.IdentityBroker.GoogleAgent.UserUpdate(IConnectorEntity[] entities, IEntitySchema schema, IDictionary`2 matchingEntities, DictionaryCache`2 renamedEntities, Boolean singleUpdate)
   at Unify.Product.IdentityBroker.GoogleAgentNotificationDecorator.UserUpdate(IConnectorEntity[] entities, IEntitySchema schema, IDictionary`2 matchingEntities, DictionaryCache`2 renamedEntities, Boolean singleUpdate)
   at Unify.Product.IdentityBroker.GoogleUserConnector.ModifyAnchor(MultiKeyValue oldKey, MultiKeyValue newKey)
   at Unify.Product.IdentityBroker.EventNotifierModifyAnchorConnectorDecorator.ModifyAnchor(MultiKeyValue oldKey, MultiKeyValue newKey)
   at Unify.Product.IdentityBroker.Adapter.<>c__DisplayClass1e.<UpdateEntities>b__12(IEntity providedEntity)
   at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
   at Unify.Product.IdentityBroker.Adapter.UpdateEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& connectorEntities)
   at Unify.Product.IdentityBroker.Adapter.UpdateEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.UpdateEntity(IAdapterEntity entityToSave)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.UpdateEntity(IAdapterEntity entityToSave)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportUpdate(IAdapter adapter, IAdapterEntitySaveChange pendingUpdate, IDictionary`2 getEntityCache)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)
   at SyncInvokeExportChanges(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
   at Unify.Product.IdentityBroker.IdentityBrokerManagementAgentProxy.ExportEntry(ModificationType modificationType, String[] changedAttributes, CSEntry csentry)

Hi Adam

I have seen that when in async mode and if import-after-export is enabled , it uses full import profile.

Is there any advantage to that? Or if delta import exists (sharedcontacts) that should be used (and hopefully for users in future) which can make it faster to then do Delta Import into FIM?

Just a query I had and your thoughts on it

When saving a Relation.Group.dn transformation (which has/should have upgraded from IDB40:Multivalue Relationship Distinguished Name Generator to IDB41:Group), the following error appears in the logs:

Request to update a selected transformation in a selected adapter.
Request to update the bbb7ffdf-d50d-4e2d-a78f-e56eb009eef3 transformation in the ba1a8cf9-e7af-42dd-b53f-34c0ca938221 adapter errored with message: Transformation of type Relation.Group.dn is not known by the adapter engine. Verify it has been added correctly to the list of known transformation generators.. Duration: 00:00:00.0156002
Error details:
System.ArgumentException: Transformation of type Relation.Group.dn is not known by the adapter engine. Verify it has been added correctly to the list of known transformation generators.
 at Unify.Product.IdentityBroker.AdapterEngine.<>c__DisplayClass7c.<UpdateTransformation>b__7b()
 at Unify.Product.IdentityBroker.AdapterEngine.<>c__DisplayClassd9.<ConfigurationChange>b__d8()
 at Unify.Framework.ExtensionMethods.WaitOnMutex(Mutex mutex, Action work)
 at Unify.Product.IdentityBroker.AdapterEngine.UpdateTransformation(Guid adapterId, IAdapterEntityTransformationConfiguration transformationConfiguration)
 at Unify.Product.IdentityBroker.AdapterEngineNotifierDecorator.UpdateTransformation(Guid adapterId, IAdapterEntityTransformationConfiguration transformationConfiguration)
 at Unify.Product.IdentityBroker.AdapterEngineAccessor.UpdateTransformation(Guid adapterId, IAdapterEntityTransformationConfiguration transformationConfiguration)
 at Unify.Product.IdentityBroker.IdentityServiceCollector.UpdateTransformation(Guid adapterId, AdapterEntityTransformationConfiguration configuration)
 at SyncInvokeUpdateTransformation(Object , Object[] , Object[] )
 at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
 at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
 at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
 at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
 at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

It looks like the upgrade is capable of reading, but is not using the new name when saving.

GoogleApps export errors not showing in FIM

In the error logs

20150829,17:27:09,UNIFY Identity Broker,Google Agent,Error,"Unify.IdentityBroker.Communicator.Google:
System.Exception: A Google API exception was thrown for call Users.Aliases.Insert with message ""Google.Apis.Requests.RequestError
Invalid Input: alias_email [400]
Errors [
	Message[Invalid Input: alias_email] Location[ - ] Reason[invalid] Domain[global]
]
"". See inner exception for details. Processing continued: True. ---> Google.GoogleApiException: Google.Apis.Requests.RequestError
Invalid Input: alias_email [400]
Errors [
	Message[Invalid Input: alias_email] Location[ - ] Reason[invalid] Domain[global]
]
   at Google.Apis.Requests.ClientServiceRequest`1.Execute()
   at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetry[TResult](String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
   --- End of inner exception stack trace ---
   at Unify.Product.IdentityBroker.GoogleAgent.<>c__DisplayClassb6`1.<BackoffRetry>b__ae()
   at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)",Normal
20150829,17:27:09,UNIFY Identity Broker,Connector,Information,"Update entities to connector completed.
Update entities [Count:1] to connector gappsUserConnector reported 1 entities saved. Duration: 00:00:03.8803880",Normal
20150829,17:27:10,UNIFY Identity Broker,Connector,Information,"Request to update entity to connector.
Request to update entities [Count:1] to connector gappsUserConnector.",Normal
20150829,17:27:13,UNIFY Identity Broker,Google Agent,Error,"Unify.IdentityBroker.Communicator.Google:
System.Exception: A Google API exception was thrown for call Users.Aliases.Insert with message ""Google.Apis.Requests.RequestError
Invalid Input: alias_email [400]
Errors [
	Message[Invalid Input: alias_email] Location[ - ] Reason[invalid] Domain[global]
]
"". See inner exception for details. Processing continued: True. ---> Google.GoogleApiException: Google.Apis.Requests.RequestError
Invalid Input: alias_email [400]
Errors [
	Message[Invalid Input: alias_email] Location[ - ] Reason[invalid] Domain[global]
]
   at Google.Apis.Requests.ClientServiceRequest`1.Execute()
   at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetry[TResult](String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
   --- End of inner exception stack trace ---
   at Unify.Product.IdentityBroker.GoogleAgent.<>c__DisplayClassb6`1.<BackoffRetry>b__ae()
   at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)",Normal
20150829,17:27:13,UNIFY Identity Broker,Connector,Information,"Update entities to connector completed.

In FIM Export

On Import no error of "exported-changes-not-reimported"

Extensions setting

Connector is in Sync mode

Leads me to believe it can be perhaps missing other error messages as well

Everytime we edit the sapAgent it does not remember the password nor does it show the ****** in it

We have to feed it in every time.

Does not happen with other Agents

Can this be fixed please.