Thread
#best-practices-coordination-plane
    Stéphan Taljaard

    Stéphan Taljaard

    3 weeks ago
    Hi I have a general Prefect 1.0 workflow question... Calling
    flow.register(..., _*idempotency_key=flow.serialized_hash()*_, ...)
    , only bumps the actual flow version if the hash changes. As expected. Bundling this with flow-based storage (instead of pickle-based), it should work well in CICD processes. It is then only required to bump the flow version if metadata changes, allowing behaviour changes inside tasks without having to re-register the flow. Let's say that you do change the innards of a task, the flow metadata does not change. Thus the serialized hash does not change. If you then run
    flow.register(..., idempotency_key=flow.serialized_hash(), ...)
    :1. A new flow script file will be built and pushed to storage (e.g. my GCS Bucket) (because the default value for build=True). 2. The flow version does not get bumped, as expected. However, when creating a flow run, it still uses the previously uploaded flow script from GCS, not the new one just created (because the unchanged idempotency key prevents pointing to the new file in storage). Thus, you either have to: • force a re-registration anyway (e.g. by not specifying a idempotency key) • "manually" upload a new flow script file into Storage, with the same name as the previous one, replacing the old one. This does not feel as automated as reading the docs makes it sound like to be? Am I missing something?
    Anna Geller

    Anna Geller

    3 weeks ago
    I agree that
    flow.register
    does not handle it well in CI/CD, but that's the main reason you ideally should use the CLI in CI/CD:
    prefect register -p path/to/flows/
    and this will only register flows that changed, no extra magic with idempotency keys required -- Prefect automatically does that for you there Also, in Prefect 2.0 this is way easier and you can see an example of CI/CD here https://github.com/anna-geller/dataflow-ops/blob/main/.github/workflows/main.yaml
    Stéphan Taljaard

    Stéphan Taljaard

    3 weeks ago
    Thanks; I'll have a look at the CLI code & behaviour (vs.
    flow.register
    ) again. I used
    flow.register
    from inside my own registration CLI script, that automatically assigns flow Storage and RunConfigs if not set, allowing flows to be set up cleanly, without their authors having to worry about "DevOps-y" stuff
    Anna Geller

    Anna Geller

    3 weeks ago
    Gotcha. I wonder whether it makes sense to work on the deployment story for Prefect 2.0 instead? given that 2.0 is the default, perhaps this would be more beneficial -- but up to you, Stephan 🙌