0
Planned

Empty String behaviour change from IdB5 upgrade for JadeStar

Bob Bradley 5 years ago in UNIFYBroker/Fusion5 JadeStar updated by anonymous 4 years ago 6

We are seeing unwanted/impacting behavioural variations in the IdB5 adapter data when compared with the legacy IdB4 adapter for the same JadeStar record.

When values are missing in the connector they are (correctly) not being surfaced in the IdB4 adapter, but are being surfaced as empty strings in IdB5.

Here is an example of the problem, as exposed for employee 500015 via LDP:

ld = ldap_open("localhost", 389);
Established connection to localhost.
Retrieving base DSA information...
Getting 1 entries:
Dn: (RootDSE)

-----------
res = ldap_simple_bind_s(ld, 'MIM_ReadWrite', <unavailable>); // v.3
Authenticated as: 'MIM_ReadWrite'.
-----------
Expanding base 'CN=500015,OU=JadeStar,DC=IdentityBroker'...
Matched DNs: CN=500015,OU=JadeStar,DC=IdentityBroker
Getting 1 entries:
Dn: CN=500015,OU=JadeStar,DC=IdentityBroker
CellphoneCountryCode: <ldp: binary="" blob="" 0="" bytes="">; 
CellphoneNumber: <ldp: binary="" blob="" 0="" bytes="">; 
CellphonePrefix: <ldp: binary="" blob="" 0="" bytes="">; 
DDiCountryCode: +64; 
DDiNumber: 222 4456; 
DDiPrefix: 4; 
DepartmentCode: 260526; 
DepartmentDescription: External Retail Network; 
DepartmentJadeCode: 1476; 
DeskNumber: <ldp: binary="" blob="" 0="" bytes="">; 
DivisionCode: SAS; 
DivisionDescription: Sales and Service; 
DivisionJadeCode: 30; 
DivisionSubCode: RETAIL CHANGE; 
DivisionSubDescription: Retail Change and Agencies; 
DivisionSubJadeCode: 225; 
EffectiveDate: 20190601000000.000Z; 
EmailAddress: <ldp: binary="" blob="" 0="" bytes="">; 
EmployeeNumber: 500015; 
EmployeeStatus: <ldp: binary="" blob="" 0="" bytes="">; 
EmployeeType: Temporary; 
EmploymentEndDate: 25000101000000.000Z; 
EmploymentStartDate: 20190601000000.000Z; 
ExpiryDate: 25000101000000.000Z; 
Extension: <ldp: binary="" blob="" 0="" bytes="">; 
FaxCountryCode: <ldp: binary="" blob="" 0="" bytes="">; 
FaxNumber: <ldp: binary="" blob="" 0="" bytes="">; 
FaxPrefix: <ldp: binary="" blob="" 0="" bytes="">; 
FirstName: Indiana; 
Function: <ldp: binary="" blob="" 0="" bytes="">; 
Initials: IM; 
JobFamily: <ldp: binary="" blob="" 0="" bytes="">; 
Level: <ldp: binary="" blob="" 0="" bytes="">; 
LocationCode: WN 20 Customhouse Quay; 
Manager: CN=852206,OU=JadeStar,DC=IdentityBroker; 
MiddleName: <ldp: binary="" blob="" 0="" bytes="">; 
objectClass: employee; 
OccupancyEndDate: 25000101000000.000Z; 
OccupancyStartDate: 20190601000000.000Z; 
OccupancyType: Std; 
OID: 7878.93689; 
OrganisationLevel: 0; 
OU: JadeStar; 
PositionCode: K2A-0012; 
PositionName: Agent Manager - Kiwibank Thorndon Sth; 
PositionOccupancyReportsToEmployee: 852206; 
PositionOccupancyReportsToName: Andrew Holford; 
PreferredName: Indiana; 
PrimaryOccupancyIndicator: TRUE; 
PrimaryPositionIndicator: FALSE; 
ReportsToEmployee: 852206; 
ReportsToName: Andrew Holford; 
ReportsToPositionCode: K2-4666; 
ReportsToPositionName: Commercial and Contracts Manager; 
StandardHoursFortnight: 80.00; 
subschemaSubentry: CN=JadeStar,cn=schema; 
Surname: Manager; 
TeamCode: EXT RETAIL NETWORK; 
TeamDescription: External Retail Network; 
TeamJadeCode: 10837; 
Title: <ldp: binary="" blob="" 0="" bytes="">; 
UserID: <ldp: binary="" blob="" 0="" bytes="">; 

-----------
</ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></ldp:></unavailable>

The above record shows "0 bytes" for a large number of attributes, as opposed to no value at all (NULL) which is what both myself and the legacy FIM rules extensions had expected.

The FIM data imported in LDIF format from IdB4, however, shows no data in these fields at all (i.e. correctly interpreting null values in the adapter).

dn: CN=500015
objectClass: person
IdBID: 65b70aac-4666-4396-9b95-0bcb33803c53
FaxCountryCode: 
EmailAddress: 
LocationCode: WN 20 Customhouse Quay
ReportsToEmployee: 852206
Initials: IM
DDiNumber: 222 4456
PreferredName: Indiana
EmployeeNumber: 500015
EmploymentStartDate: 2019-06-01T00:00:00.000
Extension: 
FaxNumber: 
ReportsToName: Andrew Holford
MiddleName: 
CellphonePrefix: 
FaxPrefix: 
EmployeeStatus: 
CellphoneCountryCode: 
OID: 7878.93689
Title: 
CellphoneNumber: 
DDiCountryCode: +64
Surname: Manager
FirstName: Indiana
DeskNumber: 
EmploymentEndDate: 2500-01-01T00:00:00.000
DDiPrefix: 4
UserID: 
EmployeeType: Temporary
DepartmentCode: 260526
DepartmentJadeCode: 1476
DivisionCode: SAS
DivisionJadeCode: 30
DivisionSubCode: RETAIL CHANGE
DivisionSubJadeCode: 225
OccupancyEndDate: 2500-01-01T00:00:00.000
OccupancyStartDate: 2019-06-01T00:00:00.000
OccupancyType: Std
PositionCode: K2A-0012
PrimaryOccupancyIndicator: True
PrimaryPositionIndicator: False
PositionOccupancyReportsToEmployee: 852206
PositionOccupancyReportsToName: Andrew Holford
StandardHoursFortnight: 80.00
TeamCode: EXT RETAIL NETWORK
TeamJadeCode: 10837
DepartmentDescription: External Retail Network
DivisionDescription: Sales and Service
DivisionSubDescription: Retail Change and Agencies
EffectiveDate: 2019-06-01T00:00:00.000
ExpiryDate: 2500-01-01T00:00:00.000
Function: 
JobFamily: 
Level: 
OrganisationLevel: 0
PositionName: Agent Manager - Kiwibank Thorndon Sth
ReportsToPositionCode: K2-4666
ReportsToPositionName: Commercial and Contracts Manager
TeamDescription: External Retail Network
Manager: CN=852206

Given the problematicdata includes fields such as CellphoneCountryCode from the base connector (i.e. untransformed), can the above behaviour be traced back to a problem with the IdB5 version of the JadeStar connector that can be easily corrected in the one place please?

TIA

Under review

Hi Bob

Can you provide your extensibility configuration for both 4.1 and 5.3?

How do these values appear in the adapter entity viewer and does this differ between versions?

Testing locally (not against JadeStar, of course) I cannot find an input value which results in the output you are seeing. Could you capture this request in a network trace?

Beau - after finally getting over all the script syntax issues in order to run a transform for all my attributes, I ended up back where I started - no net change in either MIM nor the LDP output.

Here's my PS transform in case you can spot any issue ... but no change in the result, and the logged event details follow.

Function Get-EmptyStringAsNull (
    [string]$testValue
) {
    if($testValue -ne $null -and $testValue.Length -eq 0) { 
        $testValue = $null 
    };
    return $testValue;
}

# Loop through each input entity
foreach ($entity in $entities)
{
    $msg = "";
    if (($entity['EmployeeNumber'].Value) -eq '500015') {
        $msg = "EmployeeNumber: $($entity['EmployeeNumber'].Value); "
        $msg += "Before: UserID IsNull: $($entity['UserID'].Value -eq $null); "
    }

    $entity['CellphoneCountryCode'] = (Get-EmptyStringAsNull -testValue ($entity['CellphoneCountryCode'].Value));
    $entity['CellphoneNumber'] = (Get-EmptyStringAsNull -testValue ($entity['CellphoneNumber'].Value));
    $entity['CellphonePrefix'] = (Get-EmptyStringAsNull -testValue ($entity['CellphonePrefix'].Value));
    $entity['DDiCountryCode'] = (Get-EmptyStringAsNull -testValue ($entity['DDiCountryCode'].Value));
    $entity['DDiNumber'] = (Get-EmptyStringAsNull -testValue ($entity['DDiNumber'].Value));
    $entity['DDiPrefix'] = (Get-EmptyStringAsNull -testValue ($entity['DDiPrefix'].Value));
    $entity['DepartmentCode'] = (Get-EmptyStringAsNull -testValue ($entity['DepartmentCode'].Value));
    $entity['DepartmentDescription'] = (Get-EmptyStringAsNull -testValue ($entity['DepartmentDescription'].Value));
    $entity['DepartmentJadeCode'] = (Get-EmptyStringAsNull -testValue ($entity['DepartmentJadeCode'].Value));
    $entity['DeskNumber'] = (Get-EmptyStringAsNull -testValue ($entity['DeskNumber'].Value));
    $entity['DivisionCode'] = (Get-EmptyStringAsNull -testValue ($entity['DivisionCode'].Value));
    $entity['DivisionDescription'] = (Get-EmptyStringAsNull -testValue ($entity['DivisionDescription'].Value));
    $entity['DivisionJadeCode'] = (Get-EmptyStringAsNull -testValue ($entity['DivisionJadeCode'].Value));
    $entity['DivisionSubCode'] = (Get-EmptyStringAsNull -testValue ($entity['DivisionSubCode'].Value));
    $entity['DivisionSubDescription'] = (Get-EmptyStringAsNull -testValue ($entity['DivisionSubDescription'].Value));
    $entity['DivisionSubJadeCode'] = (Get-EmptyStringAsNull -testValue ($entity['DivisionSubJadeCode'].Value));
    $entity['EmailAddress'] = (Get-EmptyStringAsNull -testValue ($entity['EmailAddress'].Value));
    $entity['EmployeeStatus'] = (Get-EmptyStringAsNull -testValue ($entity['EmployeeStatus'].Value));
    $entity['EmployeeType'] = (Get-EmptyStringAsNull -testValue ($entity['EmployeeType'].Value));
    $entity['Extension'] = (Get-EmptyStringAsNull -testValue ($entity['Extension'].Value));
    $entity['FaxCountryCode'] = (Get-EmptyStringAsNull -testValue ($entity['FaxCountryCode'].Value));
    $entity['FaxNumber'] = (Get-EmptyStringAsNull -testValue ($entity['FaxNumber'].Value));
    $entity['FaxPrefix'] = (Get-EmptyStringAsNull -testValue ($entity['FaxPrefix'].Value));
    $entity['FirstName'] = (Get-EmptyStringAsNull -testValue ($entity['FirstName'].Value));
    $entity['Function'] = (Get-EmptyStringAsNull -testValue ($entity['Function'].Value));
    $entity['Initials'] = (Get-EmptyStringAsNull -testValue ($entity['Initials'].Value));
    $entity['JobFamily'] = (Get-EmptyStringAsNull -testValue ($entity['JobFamily'].Value));
    $entity['Level'] = (Get-EmptyStringAsNull -testValue ($entity['Level'].Value));
    $entity['LocationCode'] = (Get-EmptyStringAsNull -testValue ($entity['LocationCode'].Value));
    $entity['MiddleName'] = (Get-EmptyStringAsNull -testValue ($entity['MiddleName'].Value));
    $entity['OccupancyType'] = (Get-EmptyStringAsNull -testValue ($entity['OccupancyType'].Value));
    $entity['OID'] = (Get-EmptyStringAsNull -testValue ($entity['OID'].Value));
    $entity['OrganisationLevel'] = (Get-EmptyStringAsNull -testValue ($entity['OrganisationLevel'].Value));
    $entity['PositionCode'] = (Get-EmptyStringAsNull -testValue ($entity['PositionCode'].Value));
    $entity['PositionName'] = (Get-EmptyStringAsNull -testValue ($entity['PositionName'].Value));
    $entity['PositionOccupancyReportsToEmployee'] = (Get-EmptyStringAsNull -testValue ($entity['PositionOccupancyReportsToEmployee'].Value));
    $entity['PositionOccupancyReportsToName'] = (Get-EmptyStringAsNull -testValue ($entity['PositionOccupancyReportsToName'].Value));
    $entity['PreferredName'] = (Get-EmptyStringAsNull -testValue ($entity['PreferredName'].Value));
    $entity['ReportsToEmployee'] = (Get-EmptyStringAsNull -testValue ($entity['ReportsToEmployee'].Value));
    $entity['ReportsToName'] = (Get-EmptyStringAsNull -testValue ($entity['ReportsToName'].Value));
    $entity['ReportsToPositionCode'] = (Get-EmptyStringAsNull -testValue ($entity['ReportsToPositionCode'].Value));
    $entity['ReportsToPositionName'] = (Get-EmptyStringAsNull -testValue ($entity['ReportsToPositionName'].Value));
    $entity['StandardHoursFortnight'] = (Get-EmptyStringAsNull -testValue ($entity['StandardHoursFortnight'].Value));
    $entity['Surname'] = (Get-EmptyStringAsNull -testValue ($entity['Surname'].Value));
    $entity['TeamCode'] = (Get-EmptyStringAsNull -testValue ($entity['TeamCode'].Value));
    $entity['TeamDescription'] = (Get-EmptyStringAsNull -testValue ($entity['TeamDescription'].Value));
    $entity['TeamJadeCode'] = (Get-EmptyStringAsNull -testValue ($entity['TeamJadeCode'].Value));
    $entity['Title'] = (Get-EmptyStringAsNull -testValue ($entity['Title'].Value));
    $entity['UserID'] = (Get-EmptyStringAsNull -testValue ($entity['UserID'].Value));

    if (($entity['EmployeeNumber'].Value) -eq '500015') {
        $msg += "After: UserID IsNull: $($entity['UserID'].Value -eq $null); "
        Write-EventLog -LogName Application -Source 'FIMSynchronizationService' -EntryType Information -EventId 2004 -Message $msg
    }
}

Logged event for employee 500015:

EmployeeNumber: 500015; Before: UserID IsNull: False; After: UserID IsNull: False; 

Beau - the workaround has finally worked - my problem was the function I had declared was flawed in its parameter use.  The corrected function used a new internal $result variable:

Function Get-EmptyStringAsNull (
    [string]$testValue
) {
    $result = $testValue
    if($testValue -ne $null -and $testValue.Length -eq 0) { 
        $result = $null 
    };
    return $result;
}

I accept the above work-around is going to be satisfactory for our EOFY deadline, but I have concerns that this problem has only been identified now, several years after IdB v5.05 for MIM was first released (2016-04-07).

As I explained on the call today, I believe the IdB4 behaviour for this data was already 100% correct, and the IdB5 urgently needs to be brought into line with that.  While I can't imagine any use case where an empty string value should ever be imported to MIM in any way other than NULL.

I also appreciate that given this has potentially been in place for up to 3 years at some MIM sites, and therefore to change this behaviour without necessary impact assessment on these sites would be undermine our duty of care.  To overcome this problem a potential way of softening this behaviour would be to introduce a configuration flag in Broker v5 for MIM to support both options, including backwards compatibility for IdB5 (treat '' as '') as well as backwards compatibility for IdB4 (treat '' as null).

P.S. it is the original SQL MA which has the default behaviour checked to convert empty strings to nulls:

SQL MA Options

Planned

Thanks, Bob. Added this to the backlog

+1

Here's a PowerShell transformation for correcting this behaviour on all string attributes of all entities, independent of attribute names:

function Optimize-BlankStringAttribute {
    param(
        $Entity,
        $AttributeName
    )

    if ($Entity[$attributeName].ValueType -eq "String" -and $Entity[$attributeName].Value -eq "") {
        $Entity[$attributeName] = $null
        #$Logger.LogInformation("Converting blank attribute $attributeName to NULL")
    }
}

function Optimize-Entity {
    param(
        $Entity
    )

    foreach ($attributeName in $Entity.Key.ValueName) {
        Optimize-BlankStringAttribute -Entity $Entity -AttributeName $attributeName
    }
}


foreach ($entity in $entities) {
    Optimize-Entity -Entity $entity
}