0
Fixed

Changes are not picked up on relational transformations where names are changed.

Tony Sheehy 7 years ago • updated by anonymous 3 years ago 8

A relational transformation from a connector, with a field A, joined into an adapter and called B through the column mapper will not correctly register changes.

This is because change detection uses the connector context of the relational connector, and in this context a field B does not exist.

Namely, the context comes through the EntityChangeReportGenerator through the connectorEntityDictionary, which later finds itself on the validRightEntityPairs of the RelationalTransformationContribution, which request the non-existant RightKey.

Affected Versions:
Fixed by Version:

I need to look at the required changes for this issue.

This is going to require to greater changes to justify going into RC1.

How-To fix this:

The information in the issue should be sufficient to begin investigation on fix.

ChangeDetectionImportAllJob
RepositoryChangeDetectionWorkerBase
ConnectorEntityChangeProcessor
SaveChangedEntitiesTransformationUnit - ignore, it saves the Connector entities
ChangeReportProcessor
-> ProcessCurrentReport
ChainedTransformationChangeProcessor, BaseConnectorAdapterTransformationChangeProcessor, ?
-> ProcessChangeReport

BaseConnectorAdapterTransformationChangeProcessor
-> Is _Contribution.FindBaseEntities not bringing through the changed entity?
-> Is _BaseTransformationFactory.DetermineChangeTimes not finding the timestamps for the changed entity?

JoinTransformationFactory
-> AddChainedChangeDetection - adds a ChainedTransformationChangeProcessor for the transformation
-> DetermineChangeTimes - this method might be the one being called from BaseConnectorAdapterTransformationChangeProcessor
-> ApplyChangeDetectionColumnInformation - this might be the one that adds the contribution for BaseConnectorAdapterTransformationChangeProcessor FindBaseEntities

Summary of results and resolution
Unit Test : Spent some time trying to reproduce the issue in unit tests following other similar unit tests. Ultimately this was not successful in reproducing the issue. See JoinTransformationColumnMappingTestFixture in Adapter project for implementation
UI Test : Sent time configuring CSV connectors for left and right hand side of join on IdB 4.1.1 after some time analysing the data flow and debugging the UI the issue could not be reproduced.

Finally we tested this on a v4.0.1 install configuring the exact config and the issue was reproduced. This proves that version 4.1.1 has fixed this issue.
Test Procedure:
1) Create 2 connectors based on the CSV content below

LeftSide.csv
ForeignKey,testfield
ForeignKey,test Value
RightSide.csv
ForeignKey,fieldB
ForeignKey,fieldBValue

2)Create an adapter that joins the right side to left as the base connector.
Mapping "FieldB" to a nonexistant "FieldA" on LeftSide connector.

3) Change the value for "FieldB" on RHS and do an full import on connector.

4) There should be a change registered at the DB level (possibly a pending change in the adapter)
If there is no change registered in the changes table then this means this issue has been reproduced.

JoinTransformationColumnMappingTestFixture needs a few minor updates:

  • Missing class Xml comment;
  • Comment on RelatioalTransformationChangedNames not complete;
  • RelatioalTransformationChangedNames bad spelling;
  • CreateRelationalTransformation unused, is it there for a reason for keeping it?
  • Other items next to the scrollbar to be looked at (ReSharper warnings);
  • Run the code cleanup, it'll sort out:
    • var used without context to let me know what the objects are;
    • Whitespace;

I know the rest of the code has these problems, but they'll only get improved if we are all on top of them and aren't creating more.

Thanks.

Thanks for picking this up for me. Another good habit I need to develop inline with development processes here.. I've pushed the changes to bitbucket.. Rev-2144