0
Fixed

Duplicate address error when adding members to Google Groups

Hayden Gray 5 months ago in UNIFYBroker/Google Apps updated by Beau Harrison (Senior Software Engineer) 3 months ago 10

Hi All,

I am currently assisting a client with an issue in UNIFYBroker, whereby a Google Group connector, is reporting a duplicate account error for some users when trying to added them to google groups. We can inspect the groups manually in google and these groups definitely do not contain these users. Being unable to find a root cause for the issue in Broker/MIM, the client raised a ticket with Google and received the following response:

Thank you for contacting Google Cloud Support. I understand that you are not being able to add an address to one of your Groups as you are getting a message that the address has already been to the Group.

I reviewed the information you provided on the chat and did the checks we do in this cases. The reason you are not able to add the external address is because the address you are trying to add is either an alias, a contact or a recovery address for a Google account. The system has the Google account already associated to the Group and since the system already knows the main account is already added it does not let you add it again. In this case you will need to contact the external user and ask for an alternate address they may have and check if that address is already on the Group. The other address is already receiving the messages sent to the Group.

Not being able to find a solutions around this within Broker and MIM I was hoping you may be able to take a look and see if you can spot anything we may be missing here.

This issue has quite a big of attention within the client environment as key people are missing out on important Covid-19 information. The current work around is listed in Google's response, however this often takes weeks to remediate.

Let me know if you need any further information.

Thanks

Affected Versions:
Fixed by Version:
Under review

Hi Hayden

After discussing this, it was decided that as a temporary solution we would implement the changes we talked about last week. However as I've been doing this work, I've noticed some things that do not add up.

Firstly, my understanding it that this issue was caused by adding an account or email being added to a group when another account has that email as their recovery address. I have not been able to replicate this. I've tried both internal and external members, as well as adding the recovery address both before and after the other account. Perhaps there is a setting somewhere for disallowing this that I don't have enabled in my dev environment? Can you try and find more information about this events that cause this issue? Is it actually a recovery relationship, and if so what order are the members added and is one of them an external member or are both internal members?

The other dependency is the claim that the duplicate member errors are causing other accounts to not be placed into groups if they would have been processed after the failing one. Last week I incorrectly stated that that group member processing would halt if an error occurred part way though. Digging into the code this week I've realised this is not the case, the error is just logged. Sorry for the confusion. Are there any documented cases of members not being correctly added to group without a logged error, or is this aspect of the issue based on the bad information I provided?

Finally, while debugging, I came across an issue that could cause group connector imports not properly populate the internal membership fields in certain situations. Unsure if it could have contributed to this issue, but here's a patch anyway.

Unify.IdentityBroker.Communicator.Google.dll

Hi Beau,

Sorry for the delay in testing this patch, this one appears to have brought in a large number of changes to external members. Looking at a few it appears to be delete/adding a bunch of records (Not all) based on cases. For example: deleting "Test.User@test.com" and adding "test.user@test.com". When exporting these changes IDB provides us with the following error:

Unify.IdentityBroker.Communicator.Google:
System.Exception: A Google API exception was thrown for call Members.External.Insert with message "Google.Apis.Requests.RequestError
Member already exists. [409]
Errors [
Message[Member already exists.] Location[ - ] Reason[duplicate] Domain[global]
]
". See inner exception for details. Processing continued: True. Retries: 1. ---> Google.GoogleApiException: Google.Apis.Requests.RequestError
Member already exists. [409]
Errors [
Message[Member already exists.] Location[ - ] Reason[duplicate] Domain[global]
]

at Google.Apis.Requests.ClientServiceRequest`1.Execute()
at Unify.Product.IdentityBroker.GoogleAgent.BackoffRetry[TResult](String logEvent, Boolean throwExceptions, Func`1 request, Action newClient, TResult& result, Int32 retries, Boolean retryOnInvalidValue)
--- End of inner exception stack trace ---
at Unify.Product.IdentityBroker.GoogleAgent.<>c__DisplayClass69_4`1.b__2()
at Unify.Product.IdentityBroker.GoogleAgent.ThrowIfPrimaryCall(Boolean primaryCall, Action throwException)

Looking at the data further, it appears only some groups performing delete/adds are failing with the above error. Other groups seem to be delete/adding in the same way as above and not throwing errors. I'll start the export again and see what happens if I just let it run. So far I stopped it at about 500/2500 updates. At that stage there was about 50 erroring groups.

I have also just tried clearing the connector and adapter as it is seemingly complaining about a duplicate key. And also manually searching the database for a duplicate key value and after trying both, the error still shows and I couldn't find anything in the DB.

Hi Hayden

If I recall correctly, this was the patch to allow group membership changes to continue processing even when one fails, so I'm not surprised you're seeing more activity. From what I can tell, the API error is a result of two issue:

  • Google return a success response to the remove group member request, but the actual processing of this change takes time in their back-end
  • We consider the same email with different case to be not equal, but Google does.

Basically, we try to remove a group member, Google says it's removed, and then we immediately try to add the same (according to Google) member back, except the original one is still there because the removal process takes time in their back-end.

As you saw, rerunning the export fixed this, as by that time Google processed the removal requests. Whether this is an actual issue depends on what happens if we add a group member and Google changes the case. ie we add "test.user@test.com" and Google changes this to "Test.User@test.com". This would result in a constant cycle of removing and re-adding.

The other "same key" error can be ignored. It's a known issue with the Broker portal where that error is thrown making requests to the Broker web service, seemingly at random. Minimise the effect by closing the portal when it's not in use, which I'd recommend doing anyway to avoid the overhead of its status update requests.

Hi Beau,

Thanks for clarifying that. A re-occurrence hasn't happened since the ticket was raised and since the patch has been in things have been looking good.

On the "same key" error, it seems to be happening consistently for the last week on the two user connector for both the client and myself. Even after restarting the service and doing an iisreset. Basically I just want to make sure there isn't a possibility of a duplicate key or anything. Is it safe to say that there isn't a duplicate key in these connectors base on what you've seen previously?

Thanks

Sorry, I misread your last post. Somehow missed the part where it happened on the entity search and jumped to the wrong conclusions.

The good news is it's not related to the connector entities, nor is it related to the GA patch. It's a bug in the entity search API when filtering a connector/adapter whose schema has more than one field with the same name. The only valid scenario where this is possible should be with grouped names (i.e. GroupA:Name, GroupB:Name) as Broker shouldn't even start otherwise. Is this the case for the Google connectors causing this issue?

Working on a patch for this, will be ready tomorrow.

It doesn't look like the patch is being loaded. Check that:

  • the dll has been placed in <installDir>\Identity Broker\Services
  • that folder doesn't contain any other version of that file, such as an older patch, even if it's been renamed
  • the previous point is also true of the <installdir>\Identity Broker\Services\Patches directory
  • no error messages are being logged when the service starts

Sorry Beau, I thought this file needed to go in the Web/bin directory. I've put it in the Services directory now and its working great now. Thanks for your help.