Google User Settings Connector import failing

The error appears to be caused by the API abstracting away the authentication token (something I don't have control over). I've updated it to use the same retry logic as the other calls (as well as running the calls in parallel), so that the client should be recycled when it fails.

Unify.IdentityBroker.Communicator.Google.dll

Hi Adam

Just had a chance to try this out. It failed again after approximately 7 minutes (error attached).

Error Google User Settings 2.txt

This is a new error. However, as it's triggered by the exact same piece of code I'll look at it on this ticket.

Updated the retry logic to backoff retry timeouts:

Unify.IdentityBroker.Communicator.Google.dll

A new error occurred (attached)

Error Google User Settings 3.txt

I've throttled the calls so that Google doesn't have a fit. Unfortunately it touches all the files, so I've uploaded a new MSI.

Hi Adam

I have deployed this into an non-production environment using the same configuration. I have been been able to perform Full Imports on the User and Group connectors without any errors. I get the attached error on the User Settings Connector soon after the import starts.

Boyd

Error Google User Settings 4.txt

Unify.IdentityBroker.Communicator.Google.dll

That should be it now, if not try tuning down the MaxDegreeOfParallelism setting. If that also fails let me know if I can connect to a dev Google Apps environment to test myself.

Thanks.

Hi Adam

The issue is still occurring, I reduced the Parallel limit from 1000 to 100 and no change.

I have attached the Agent settings and certificates, it is a Production domain so please limit testing to Read operations.

If this Connector is using the Email Settings API it appears to be have been replaced with the Gmail API: https://developers.google.com/admin-sdk/email-settings/

Adam: Made attachments private.

I had been keeping an eye out for a replacement for the old API's... thanks for pointing it out to me.

There is no client library for it yet (https://www.nuget.org/profiles/google-apis-packages?showAllPackages=True), so I can't migrate over. It seems strange that they'd deprecate without a path forward, but it's not the first time it's happened with Google!

Rubbish Google documentation strikes again! Matthew pointed out that it's now the Gmail API.

Can you please enable the following scopes?

• https://www.googleapis.com/auth/gmail.settings.basic
• https://www.googleapis.com/auth/gmail.settings.sharing

Scopes have been autorised for both domains in the configuration. List of authorised scopes is now:

https://apps-apis.google.com/a/feeds/emailsettings/2.0/
https://www.google.com/m8/feeds/
https://www.googleapis.com/auth/admin.directory.group
https://www.googleapis.com/auth/admin.directory.group.member
https://www.googleapis.com/auth/admin.directory.group.member.readonly
https://www.googleapis.com/auth/admin.directory.group.readonly
https://www.googleapis.com/auth/admin.directory.orgunit
https://www.googleapis.com/auth/admin.directory.orgunit.readonly
https://www.googleapis.com/auth/admin.directory.user
https://www.googleapis.com/auth/admin.directory.user.alias
https://www.googleapis.com/auth/admin.directory.user.readonly
https://www.googleapis.com/auth/apps.groups.settings
https://www.googleapis.com/auth/calendar
https://www.googleapis.com/auth/gmail.settings.basic
https://www.googleapis.com/auth/gmail.settings.sharing

Thanks, that has allowed it to progress to the next error.

Please ensure delegation is set up for the service account (as per https://developers.google.com/identity/protocols/OAuth2ServiceAccount#delegatingauthority). The error I'm seeing is: Message[Delegation denied for fim@cns.catholic.edu.au] Location[ - ] Reason[forbidden] Domain[global]

Any luck with setting up delegation?

Hi Adam

I have setup the delegation in the cns.catholic.edu.au domain as per previous APIs. However I cannot successfully test in API Explorer so am unsure if it working or not. Is there any way you can test on your end?

Confirmed with someone here at Cairns Catholic Education working with Google APIs that he can use the API in code but not using API Explorer.

Perfect, thanks for that. I'll do up a build when I'm back in the office so that we can test using the connector.

Still no luck:

Google.Apis.Auth.OAuth2.Responses.TokenResponseException: Error:"unauthorized_client", Description:"Client is unauthorized to retrieve access tokens using this method."

Hi Adam

I have been working with David here at Cairns Catholic Education and he was able to get the Service Account to return the SendAs information for another user using the domain-wide delegation permissions. During this we did get the unauthorized_client error which was found to be because we were calling the incorrect service. The message seems to be triggered when a supplied scope does not exist for that service or possibly if a required scope is missing.

We did add https://www.googleapis.com/auth/gmail.labels to the authorized scope as David's existing code included this scope.

Boyd

Fantastic, that combination has allowed some items to come back. However, I'm now receiving the following error:

Change detection engine import all items failed.Change detection engine import all items for connector User Settings failed with reason One or more errors occurred.. Duration: 00:05:13.8955363Error details:System.AggregateException: One or more errors occurred. ---> Google.Apis.Auth.OAuth2.Responses.TokenResponseException: Error:"invalid_grant", Description:"Invalid email or User ID", Uri:""

I'm debugging now to try to find out whether it's a specific user causing the issue.

See updated version. The update likely isn't working due to scopes, but I didn't want to mess with your instance. Let me know the correct scopes and I'll update it.

Thanks.

Hi Adam

I get the following error when I try to start Broker after applying the update. Is there an updated DLL I need?

Service cannot be started. Unify.Framework.UnifyServiceInitializeException: The type initializer for 'Unify.Product.IdentityBroker.GoogleAgent' threw an exception. ---> System.TypeInitializationException: The type initializer for 'Unify.Product.IdentityBroker.GoogleAgent' threw an exception. ---> System.IO.FileLoadException: Could not load file or assembly 'Google.Apis, Version=1.19.0.0, Culture=neutral, PublicKeyToken=4b01fa6e34db77ab' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
at Unify.Product.IdentityBroker.GoogleAgent..cctor()
--- End of inner exception stack trace ---
at Unify.Product.IdentityBroker.GoogleAgent..ctor(INotificationMessageService messageService, IGoogleAgentInformation agentInformation, String displayName)
at Unify.Product.IdentityBroker.GoogleAgentFactory.CreateComponent(IAgentConfiguration factoryInformation)
at Unify.Product.IdentityBroker.AgentEngine.GenerateAgent(IAgentConfiguration con...

Ah sorry, thought I had mentioned on an earlier issue. The referenced assemblies were updated, so the binding redirects need updated (https://unifysolutions.jira.com/wiki/display/IDBGA50/Prerequisites). You'll also need to change the user settings key from username over to email (or clear the schema and rerun the schema provider).

Thanks.

Hi Adam

It ran for about 3 minutes and processed the first 1000 entries before it got the error below related to threads.

GMail Error.txt

Boyd

I didn't get that on my local machine. Try tuning down the parallel thread limit on the agent.

This import was successful. Will start test exports.

Hi Adam

I am getting the attached error when attempting to update the sendAs attribute which is multi-valued XML.

GMail Error 19-04-2017.txt

The changes are below, the changes are adapted from other Gmail accounts which were configured manually.

delete,"<SendAs name="""" address=""bbostock@cns.catholic.edu.au"" replyTo="""" signature="""" default=""true"" />"

The export from MIM now works without error and Identity Broker Adapter and Connector entries are updated with the exported values however the GMAIL settings are not changed in the UI and a subsequent import of the Connector shows the exported values did not apply.

If you need to test you can modify the settings for bbostock@cns.catholic.edu.au.

Thanks, will do.

There's a new release available that now makes the correct call. However, it's now failing due to scope or permissions error. Can you take a look?

Thanks

Hi Adam

I found https://www.googleapis.com/auth/gmail.settings.sharing was missing from the list of authorised scopes. I have added the scope and restarted Identity Broker but still get the error. Is this scope included in the call or does Identity Broker cache a token which needs to be deleted?

Boyd

I had missed that from the list of scopes. I've added it in:

Unify.IdentityBroker.Communicator.Google.dll

And not sure about the tokens being cached, as it's abstracted away. I don't think so though, as it now retrieves a new token for every user (as it has to impersonate each user), due to the API not working properly for a service account.