A question about flow and task versions (Prefect Orion)Hi all,
Quick intro for our use-case and how we got to the idea of maybe starting to use flow/task versioning (triggered by the latest release 2.0b8 which includes this option for tasks as well now).
The root idea stems from the fact that with CI/CD up and running, automatically deploying new versions to our prefect server, at some point we want to be able to double/sanity check that the correct version of the code is being ran in a flow (because, if flow X has a currently active flow run, and we deploy a new version for flow X, the already running flow will still have the old code and the next flow might have the new code) .
Another colleague noticed this: https://orion-docs.prefect.io/tutorials/flow-task-config/ mentioning the usage of GIT_COMMIT_SHA as a version. This seemed nifty so we tried it out. However, we had to do some manual labor to get this to work, to try and get this git commit in the env variables, which currently basically relies on our Makefile and CI/CD, so this seemed a bit cumbersome and not ideal. We managed to make it work. But as an extra we have a lot of code that we need to add to all the flows, so on top of that maybe it would be nice if somewhere we could configure a default for this version using a callback that could be specified somewhere in the task/flow source code. Because currently the only way we could come up with setting a default is by basically overriding the flow/task decorators from Prefect, but that does not feel right.Wondering if anyone has any other takes that might be a bit more clean on tracking what version of code is actually running?
2 months ago
I totally understand what you mean. The way I used to manage it in a project was to use GITHUB_SHA for the Docker image tag that gets pushed to ECR and this way, track code changes with deployment artifactslet me open an issue to investigate how we may improve the process here and provide more guidance on how to integrate this flow/task versioning with GitHub CI/CD flow@Marvin open "Provide docs/recipes on how to version flows and tasks in a CI/CD GitOps process"