Bugs
Time Offset Flag not re-evaluated when current time passes source field timestamp
My customer is failing UAT of the solution configuration because Time Offset Flags are not automatically updating when the source field timestamp is passed. There have not been any Clear Entity Changes run in this environment for many months, and entity data fields have been updated recently as part of the customer's UAT testing processes.
Example: entity ID 70cb5e8e-8a8d-48f9-a123-911a836574f4 in partition 838b79fc-a31e-4b70-bcc8-e94550b3ff57 in ACCC TEST. PostEnd, based on EndUTC, is still "No" but it should be "Yes".
Could you please check entity ID e71b989c-1a01-4542-9334-8e69c12abb6c on that same partition, which is due to see PostEnd change from "No" to "Yes" in around 15 hours from now (8/22/2023 2:00:00 PM UTC). Please confirm that it is all fine (i.e., a future change exists in the database) and then after the timestamp is passed I'll check and verify if the PostEnd value has updated automatically as it should.
Answer
Morning Adrian,
I've had a quick look and can see that there is a change registered to run at 10am UTC on the 22/08 for entity e71b989c-1a01-4542-9334-8e69c12abb6c. This may or may not be related to the field you're expecting to change though, and based on the timestamp you've provided may run earlier than you're expecting.
The issue you're describing is similar to this issue though: Time Offset Flag didn't re-evaluate when date threshold was passed / UNIFYBroker Forum / UNIFY Solutions
I had a look through the config and noticed that the fields being used by the Time Offset Flag are being created by a powershell transformation. I had a quick glance through the module and couldn't see anywhere that the Register-Contribution call was being made on the transformation schema. Have you wired up this functionality in the transformation to ensure changes are registered correctly for offset transformations?
Thanks Matt. If the change evaluates at 10am and not at 2pm then it won't be evaluated correctly (because it won't be past the transition timestamp when it evaluates). Do you have any suggestions why it's not set to evaluate at 2pm?
I briefly looked through that ticket prior to creating this one - it's a long one, and I thought it related to having run a Clear Entity Change and losing the scheduled changes. I must confess that in the end I couldn't quite get my head around exactly when Register-Contribution should be called and what it does. Could you please let me know where I should use it? If the outcome overnight is incorrect tonight then I'll put it in place and try again.
Changes are registered for the times that the contributing transformations dictate they need to be registered. In this case, there could have been a scenario under which a transformation has determined, based on current (or previous) configuration, that a change should be created for that time because at that time a date time offset flag or similar needed to be recalculated. It may have been from one of the time offset fields that has a value already and is known to recalculate at that time.
The original part of the ticket is as you suspect - where clearing pending changes will remove future dated changes, and they won't be recreated through a generate changes process (which is something we've improved for UNIFYBroker 6.0).
The feature (Register-Contribution) was added to handle a scenario where a PowerShell transformation is adding/modifying a field, which is used in a transformation that can calculate future-dated changes (such as the Time Offset Flag transformation). With a normal chain of transformations (powershell aside), each adapter tracks the chain of where its source came from. This is done so we can calculate whether or not a change is needed for an entity (for example, if field X from relational connector A is used through a chain of transformations and then to calculate a datetime offset, we need to know on import if field X has a value which would trigger a change in the future, so we can register that change in the database so it recalculates at the right time, rather than too early / late). This calculation process is called Change Detection in the Broker engine.
Traditionally, the PowerShell transformation had no way of letting the Broker engine know how entity fields were being used, so it had no way of being involved in the change detection process. This also means it broke downstream change detection - if a PowerShell transformation is outputting a field value which is then used in a Time Offset Flag transformation, the engine had no way of knowing the source field of that value to notify of future dated changes.
The Register-Contribution call allows you to register this linkage to let the engine know how to handle those scenarios. So as an example, you may have an EndDate schema field coming from the parent or relational connector. You take this EndDate field in, and use a PowerShell transformation to convert it to UTC time, placing it into an EndTimestampUTC field. That EndTimestampUTC field is then used to calculate the PostEnd field through a TimeOffsetFlag. You would use the Register-Contribution call on this field, to essentially tell the adapter that "the PostEnd field passes through this magical script and results in the EndTimestampUTC field, so if there's a change on the PostEnd field it can be used to calculate changes on the EndTimestampUTC field". The adapter can then use that linkage to know that a change on the PostEnd field which sets the date in the future can be used to calculate a change in the future based on the configuration for the TimeOffsetFlag transformation.
The configuration would look similar to this (screenshot from the linked ticket):
TL;DR:
The PowerShell transformation does not participate in the change detection process by default. This can be enabled by manually describing the fields which contribute in some way to other fields, either created by the transformation, or pre-existing.
To do this, call the Register-Contribution method in the Schema Script for each instance of one field contributing to another.
# 'fieldA' contributes to the resulting value in 'fieldB'
Register-Contribution("fieldB", "fieldA");
Manually registering field contribution isn't required in most cases and for all fields, and can normally be omitted from the schema script. The typical situations where contribution registration would be required involve a PowerShell transformation preceding a Time Offset Flag or Business Day Offset transformations, where the contribution chain of the the involved Timestamp or Date fields is required to correctly schedule future-dated changes.
Thanks Matt, I'll add Register-Contribution in accordance with your description above. In the absence of any further update here, please assume that has been successful and close this ticket.
Customer support service by UserEcho
Changes are registered for the times that the contributing transformations dictate they need to be registered. In this case, there could have been a scenario under which a transformation has determined, based on current (or previous) configuration, that a change should be created for that time because at that time a date time offset flag or similar needed to be recalculated. It may have been from one of the time offset fields that has a value already and is known to recalculate at that time.
The original part of the ticket is as you suspect - where clearing pending changes will remove future dated changes, and they won't be recreated through a generate changes process (which is something we've improved for UNIFYBroker 6.0).
The feature (Register-Contribution) was added to handle a scenario where a PowerShell transformation is adding/modifying a field, which is used in a transformation that can calculate future-dated changes (such as the Time Offset Flag transformation). With a normal chain of transformations (powershell aside), each adapter tracks the chain of where its source came from. This is done so we can calculate whether or not a change is needed for an entity (for example, if field X from relational connector A is used through a chain of transformations and then to calculate a datetime offset, we need to know on import if field X has a value which would trigger a change in the future, so we can register that change in the database so it recalculates at the right time, rather than too early / late). This calculation process is called Change Detection in the Broker engine.
Traditionally, the PowerShell transformation had no way of letting the Broker engine know how entity fields were being used, so it had no way of being involved in the change detection process. This also means it broke downstream change detection - if a PowerShell transformation is outputting a field value which is then used in a Time Offset Flag transformation, the engine had no way of knowing the source field of that value to notify of future dated changes.
The Register-Contribution call allows you to register this linkage to let the engine know how to handle those scenarios. So as an example, you may have an EndDate schema field coming from the parent or relational connector. You take this EndDate field in, and use a PowerShell transformation to convert it to UTC time, placing it into an EndTimestampUTC field. That EndTimestampUTC field is then used to calculate the PostEnd field through a TimeOffsetFlag. You would use the Register-Contribution call on this field, to essentially tell the adapter that "the PostEnd field passes through this magical script and results in the EndTimestampUTC field, so if there's a change on the PostEnd field it can be used to calculate changes on the EndTimestampUTC field". The adapter can then use that linkage to know that a change on the PostEnd field which sets the date in the future can be used to calculate a change in the future based on the configuration for the TimeOffsetFlag transformation.
The configuration would look similar to this (screenshot from the linked ticket):
TL;DR:
The PowerShell transformation does not participate in the change detection process by default. This can be enabled by manually describing the fields which contribute in some way to other fields, either created by the transformation, or pre-existing.
To do this, call the Register-Contribution method in the Schema Script for each instance of one field contributing to another.
# 'fieldA' contributes to the resulting value in 'fieldB' Register-Contribution("fieldB", "fieldA");Manually registering field contribution isn't required in most cases and for all fields, and can normally be omitted from the schema script. The typical situations where contribution registration would be required involve a PowerShell transformation preceding a Time Offset Flag or Business Day Offset transformations, where the contribution chain of the the involved Timestamp or Date fields is required to correctly schedule future-dated changes.