Stéphan Taljaard
08/31/2022, 9:30 AMflow.register(..., _*idempotency_key=flow.serialized_hash()*_, ...)
, only bumps the actual flow version if the hash changes. As expected.
Bundling this with flow-based storage (instead of pickle-based), it should work well in CICD processes. It is then only required to bump the flow version if metadata changes, allowing behaviour changes inside tasks without having to re-register the flow.
Let's say that you do change the innards of a task, the flow metadata does not change. Thus the serialized hash does not change. If you then run flow.register(..., idempotency_key=flow.serialized_hash(), ...)
:
1. A new flow script file will be built and pushed to storage (e.g. my GCS Bucket) (because the default value for build=True).
2. The flow version does not get bumped, as expected.
However, when creating a flow run, it still uses the previously uploaded flow script from GCS, not the new one just created (because the unchanged idempotency key prevents pointing to the new file in storage).
Thus, you either have to:
• force a re-registration anyway (e.g. by not specifying a idempotency key)
• "manually" upload a new flow script file into Storage, with the same name as the previous one, replacing the old one.
This does not feel as automated as reading the docs makes it sound like to be?
Am I missing something?Anna Geller
flow.register
does not handle it well in CI/CD, but that's the main reason you ideally should use the CLI in CI/CD:
prefect register -p path/to/flows/
and this will only register flows that changed, no extra magic with idempotency keys required -- Prefect automatically does that for you there
Also, in Prefect 2.0 this is way easier and you can see an example of CI/CD here https://github.com/anna-geller/dataflow-ops/blob/main/.github/workflows/main.yamlStéphan Taljaard
08/31/2022, 10:29 AMflow.register
) again.
I used flow.register
from inside my own registration CLI script, that automatically assigns flow Storage and RunConfigs if not set, allowing flows to be set up cleanly, without their authors having to worry about "DevOps-y" stuffAnna Geller