Netanel Malka
05/19/2024, 9:43 AMclass OrchestrationConf(BaseModel):
flows: OrderedDict[str, bool] = Field(
default={},
description="Ordered objects of flow names and their enable status. The order of the flows is important."
)
Then pass it using the prefect.yaml:
parameters:
conf:
flows:
asset_daily_flatten_signals: true
asset_daily_agg: true
asset_weekly_agg: true
asset_period_agg: true
But Prefect don't preserve the order of the flows, from Prefect UI:
{
"conf": {
"flows": {
"asset_daily_agg": true,
"asset_period_agg": true,
"asset_weekly_agg": true,
"asset_daily_flatten_signals": true
}
}
}
How can I preserve the order of the flows?Marvin
05/19/2024, 9:43 AMOrderedDict
is designed to maintain order, but when serialized to JSON, which is how parameters are likely passed to Prefect's UI and YAML files, order may not be preserved since JSON objects are inherently unordered.
One approach to circumvent this would be to use a list of objects or tuples instead of a dictionary because JSON arrays do preserve order. You can provide your parameters as a list of two-item lists (or tuples) where the first item is the key and the second item is the value.
Here's an updated example of how you might define it:
yaml
parameters:
conf:
flows:
- [asset_daily_flatten_signals, true]
- [asset_daily_agg, true]
- [asset_weekly_agg, true]
- [asset_period_agg, true]
Then, in your flow, you would have to parse this list and make an ordered dictionary out of it manually, if that is necessary for your use case.
If the above suggestion doesn't help, I'm more than capable—albeit less than enthusiastic—about scouring the Prefect documentation and GitHub issues further to find a better solution for preserving order for parameters passed through Prefect configurations. Please let me know if you need further assistance.