Migrate Calendar Resource connector from Atom (old) to JSON API (current v1 Admin SDK) as it is set to be stopped by Jan 2017.

Hi Adam

Can you please help me answer these questions as Google has asked me the same. I had requested higher API limits for our accounts. Currently I am chewing threw them .. probably at some rate of around 30000 accounts for 150k API Calls.. have gone through 3 accounts already and have managed just to push out 70-80k object changes

The number of users. - About 300,000
Average number of requests per day/per user (calculation describing your expected usage of the API).
Which API methods will be called and what will be the frequency?Are you polling the API changes to user resources?

Hi

Today we are seeing the following error in gapps contacts and calendar

20150602,23:59:55,UNIFY Identity Broker,Change detection engine,Error,"Change detection engine import all items failed.
Change detection engine import all items for connector gappsCalendarConnector failed with reason Stream was not readable.. Duration: 00:00:00.7488192
Error details:
System.ArgumentException: Stream was not readable.

20150602,23:59:55,UNIFY Identity Broker,Change detection engine,Error,"Change detection engine import all items failed.
Change detection engine import all items for connector gappsSharedContactConnector failed with reason Stream was not readable.. Duration: 00:00:00.6240160
Error details:
System.ArgumentException: Stream was not readable.

Our developer has tested the fim-svc credentials with oauth 2.0 and he says the credentials are fine.

We are seeing this in our DEV and QAT google environments.. but not with users..
Any idea on what is causing this?

Currently unable to import Google Group settings (https://developers.google.com/admin-sdk/groups-settings/v1/reference/groups/get), not only in Identity Broker, but in the Google test explorer.

Have confirmed that the required scope has been applied to the account.

Group.PNG shows that the Groups are behaving fine.
GroupsSettings.PNG shows the error from Google.

Google Apps Connector has an issue that when we import (with primaryEmail as key) it import users as duplicates. Those records are not duplicate in the data source, I have verified it by reviewing the data via Google App admin portal.

I have marked it as blocker as it prevents us from importing/exporting data from Google Apps and blocking our DEV and TEST activity.

Hi

Has the gappsSharedContactConnector changed?

I believe previously we had to put in
Env:
Username:
Password:

while creating the connector. Now it does not seem to be the case?

Just wanted to confirm. Its better for us if it was removed. Easier to maintain..

If that is the case.. can we do the same for calendar connector? We still have to mention domain there.. and if it can pickup from agent file, we won't have to maintain three separate Unify.Product.IdentityBroker.ConnectorEnginePlugInKey.extensibility.config.xml files for DEV / QAT / PROD (due to static domain fields)

But if calendar connector requires it for some reason then we can let it be.

We did set a bench mark for 10,000 exports/hour – which translates to about 8 exports per second.
The current speeds for exports are at 5-7 users per minute. We would expect out peak student intake to take ~24 hours to process.
I did call out very early in the piece that performance was important, but tricky to get right. No consulting has been done with Monash on the export process or results shared of IdB internal benchmarks. As it stands, this is a show stopping issue.

The XML Blob for organization in objectclass user is not working as expected. Having exported-changes-not-imported issues

Expo

Request to import all entities from connector gappsCalendarConnector.",Normal
20150618,04:51:49,UNIFY Identity Broker,Connector,Warning,"Import all entities from connector failed.
Import all entities from connector gappsCalendarConnector failed with reason A Google Data exception was thrown for call CalendarResources.RetrieveAllCalendarResources with message ""Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/"". See inner exception for details. Processing continued: False. Response: Version 3.0 is not supported.. Duration: 00:00:07.2319590
Error details:
System.Exception: A Google Data exception was thrown for call CalendarResources.RetrieveAllCalendarResources with message ""Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/"". See inner exception for details. Processing continued: False. Response: Version 3.0 is not supported. ---> Google.GData.Client.GDataRequestException: Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/ ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Google.GData.Client.GDataRequest.Execute()
— End of inner exception stack trace —
at Google.GData.Apps.AppsPropertyService.QueryExtendedFeed(Uri uri, Boolean shouldGetAllPages)
at Google.GData.Apps.CalendarResourceService.RetrieveAllCalendarResources()
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetryTResult(String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
— End of inner exception stack trace —
at Unify.Product.IdentityBroker.GoogleAgent.<>c_DisplayClassab`1.<BackoffRetry>b_a5()
at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetryTResult(String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
at Unify.Product.IdentityBroker.GoogleAgent.<CalendarGetPages>d__23.MoveNext()
at Unify.Product.IdentityBroker.GoogleCalendarConnector.<GetAllEntities>d__15.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Unify.Framework.Collections.ActionOnExceptionEnumerator`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ActionOnLast>d__19`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ProduceAutoPages>d__a`1.MoveNext()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPoolT(IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Framework.Visitor.VisitEvaluateOnThreadPoolT(IEnumerable`1 visitCollection, Action`1 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.JobBase.Run()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c_DisplayClass29.<Run>b_27()
at Unify.Framework.ActionJob.RunBase()
at Unify.Framework.JobBase.Run()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal
20150618,04:51:49,UNIFY Identity Broker,Change detection engine,Error,"Change detection engine import all items failed.
Change detection engine import all items for connector gappsCalendarConnector failed with reason A Google Data exception was thrown for call CalendarResources.RetrieveAllCalendarResources with message ""Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/"". See inner exception for details. Processing continued: False. Response: Version 3.0 is not supported.. Duration: 00:00:07.2739485
Error details:
System.Exception: A Google Data exception was thrown for call CalendarResources.RetrieveAllCalendarResources with message ""Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/"". See inner exception for details. Processing continued: False. Response: Version 3.0 is not supported. ---> Google.GData.Client.GDataRequestException: Execution of request failed: https://apps-apis.google.com/a/feeds/calendar/resource/2.0/ga-staff-dev.monash.edu/ ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Google.GData.Client.GDataRequest.Execute()
— End of inner exception stack trace —
at Google.GData.Apps.AppsPropertyService.QueryExtendedFeed(Uri uri, Boolean shouldGetAllPages)
at Google.GData.Apps.CalendarResourceService.RetrieveAllCalendarResources()
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetryTResult(String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
— End of inner exception stack trace —
at Unify.Product.IdentityBroker.GoogleAgent.<>c_DisplayClassab`1.<BackoffRetry>b_a5()
at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetryTResult(String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
at Unify.Product.IdentityBroker.GoogleAgent.<CalendarGetPages>d__23.MoveNext()
at Unify.Product.IdentityBroker.GoogleCalendarConnector.<GetAllEntities>d__15.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Unify.Framework.Collections.ActionOnExceptionEnumerator`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ActionOnLast>d__19`1.MoveNext()
at Unify.Framework.Collections.EnumerableExtensions.<ProduceAutoPages>d__a`1.MoveNext()
at Unify.Framework.Visitor.ThreadsafeVisitorEvaluator`1.Visit()
at Unify.Framework.Visitor.VisitEvaluateOnThreadPoolT(IEnumerable`1 visitCollection, Action`2 visitor, Int32 maxThreads)
at Unify.Framework.Visitor.VisitEvaluateOnThreadPoolT(IEnumerable`1 visitCollection, Action`1 visitor, Int32 maxThreads)
at Unify.Product.IdentityBroker.RepositoryChangeDetectionWorkerBase.PerformChangeDetection(IEnumerable`1 connectorEntities)
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.ImportAllChangeProcess()
at Unify.Product.IdentityBroker.ChangeDetectionImportAllJob.RunBase()
at Unify.Framework.JobBase.Run()
at Unify.Framework.DefinedScopeJobAuditTrailJobDecorator.Run()
at Unify.Product.IdentityBroker.ConnectorJobExecutor.<>c_DisplayClass29.<Run>b_27()
at Unify.Framework.ActionJob.RunBase()
at Unify.Framework.JobBase.Run()
at Unify.Framework.AsynchronousJobExecutor.PerformJobCallback(Object state)",Normal

We are seeing that googleapps MA is reporting errors in FIM in async mode.

Wanted to get more insight on why this is happening because as per our understanding there should be no error reporting in async mode.

Attached Unify Log from yesterday with loads of errors..

Attaching screenshot of errors in FIM

The error from one of them (reported in FIM)

System.Exception: Error occurred when attempting to save entity with distinguished name

CN=75fd882f-a487-4e03-9a77-5d3af811a8e1,OU=users

Error:

A Google API exception was thrown for call Users.Update with message "Google.Apis.Requests.RequestError
Entity already exists. [409]
Errors [
	Message[Entity already exists.] Location[ - ] Reason[duplicate] Domain[global]
]
". See inner exception for details. Processing continued: False.
   at Unify.Product.IdentityBroker.GoogleAgent.<>c__DisplayClassb1`1.<BackoffRetry>b__a9()
   at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)
   at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetry[TResult](String logEvent, Boolean primaryCall, Func`1 request, TResult& result, Int32 retries)
   at Unify.Product.IdentityBroker.GoogleAgent.UserUpdate(IConnectorEntity[] entities, IEntitySchema schema, IDictionary`2 matchingEntities, DictionaryCache`2 renamedEntities, Boolean singleUpdate)
   at Unify.Product.IdentityBroker.GoogleAgentNotificationDecorator.UserUpdate(IConnectorEntity[] entities, IEntitySchema schema, IDictionary`2 matchingEntities, DictionaryCache`2 renamedEntities, Boolean singleUpdate)
   at Unify.Product.IdentityBroker.GoogleUserConnector.ModifyAnchor(MultiKeyValue oldKey, MultiKeyValue newKey)
   at Unify.Product.IdentityBroker.EventNotifierModifyAnchorConnectorDecorator.ModifyAnchor(MultiKeyValue oldKey, MultiKeyValue newKey)
   at Unify.Product.IdentityBroker.Adapter.<>c__DisplayClass1e.<UpdateEntities>b__12(IEntity providedEntity)
   at Unify.Framework.Visitor.Visit[T](IEnumerable`1 visitCollection, Action`2 visitor)
   at Unify.Product.IdentityBroker.Adapter.UpdateEntities(IEnumerable`1 entities, EntityToConnectorEntityBridge[]& connectorEntities)
   at Unify.Product.IdentityBroker.Adapter.UpdateEntities(IEnumerable`1 entities)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.UpdateEntity(IAdapterEntity entityToSave)
   at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.UpdateEntity(IAdapterEntity entityToSave)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.HandleExportUpdate(IAdapter adapter, IAdapterEntitySaveChange pendingUpdate, IDictionary`2 getEntityCache)
   at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter)
   at SyncInvokeExportChanges(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
   at Unify.Product.IdentityBroker.IdentityBrokerManagementAgentProxy.ExportEntry(ModificationType modificationType, String[] changedAttributes, CSEntry csentry)