Unify.Framework.EvaluatorVisitorException: An error occurred while evaluating a task on a worker thread. See the inner exception details for information. ---> Npgsql.PostgresException: 22P05: unsupported Unicode escape sequence

Adrian Corston 6 months ago in UNIFYBroker/Plus updated by Beau Harrison (Senior Software Engineer) 6 months ago 9

UNIFYConnect instance created some new users in an AD connector, but then the next time it ran the following error appeared:

Affected Versions:
Fixed by Version:
Satisfaction mark by Adrian Corston 6 months ago

I changed the object container to exclude the newly created user by sAMAccountName and it works fine.  Further tweaking identified that it's only one particular user who causes the error (the second in my protected post above - B.D.).

The source of that user's data is the HRMS CSV file, which I edited using Notepad.  The R.H. user was a copy/paste/edit of the B.D. user, so I'm at a loss to understand how it could have different unicode characters in it.

There are no evident unicode characters in either the source CSV or the AD record.

Under review

Hi Adrian

The invalid Unicode character mentioned by the error message is the null character: \u0000. Try and figure out which field contains the null character and remove it from the connector schema. I'm not sure what AD tools available would let you see invisible Unicode characters, or if it could be done in PowerShell, but you could just systematically removing fields and try importing until it succeeds.

If you can't remove the null character, the only workaround at the moment is to not import that value. I'll dig into the more next week.

Thanks Beau, I've removed and retyped that user's entry line in the source CSV file and now all is well.

Sadly it appears I spoke too soon.  Now I've opened the LDAP filter to load all customer users (as loaded from the production LDIF files) the error is back.  I have no easy way to identify which user (of 1888) or field (of 20 or so) is affected, other than systematically updating the connector configuration to exclude/re-include fields and users until I work it out, which is likely to take quite some time.

For reference, this is what I tried:

$Users = get-aduser -filter * -Properties *
$Fields = ($Users[0] | get-member -MemberType Property).name
foreach ($User in $Users) {
foreach ($Field in $Fields) {
if ($User.$Field -ne $null) {
if ($User.$Field.ToString() -contains [char]0) {
write-host "Field $Field for user $($User.sAMAccountName) contains NUL character"

It looks like this might be happening for me when I import objectGUID as a string.

Hi Adrian, a patch has been deployed that should correct this issue. Note that the null byte is now being correctly stored in the database, they are not removed. You may need to find and sanitize these values in an adapter PowerShell transformation if you find they cause cause you any issues, depending on how and where they're used.

Thanks Beau.  I should have said this explicitly: I think the mistake here was me importing objectGUID as a string.  I've stopped doing that now.

Thank you for the patch.