Jenia Varavva
04/17/2023, 10:46 PMgs://<bucket>/<commit hash>
2. For each of the cartesian product of Flow ⅹ Env (e.g. A-prod
, B-qa
) have:
a. A separate storage block pointing to a different subdir of gs://<bucket>/
b. A docker image label pointing to the image of the same commit.
3. The release to an environment would then include: updating the storage block to point to a different dir + updating the container image tag. Each deployment would also have an infra_override
pointing it at its specific image tag.
A couple of concerns I’m having are: storing all code/image versions (might be a good thing?). Maintaining potentially a large number of storage blocks (one for each of flow x env). Any thoughts?Henning Holgersen
04/18/2023, 6:05 AMPRREFECT_DEPLOY_TO
variable. If the variable isn’t set, we can assume it is a developer using their local machine, and everything can be resolved to have a dev_
prefix or something like that. We have a rule that everything in production must be deployed from the main branch, and so we have a github action to register the flows. Here, you can explicitly set the PREFECT_DEPLOY_TO
variable to prod
or something like that, and all references to storage blocks etc will be resolved without the dev_
prefix.
3. We do by default use a single docker image in multiple flows. We can of course build particular docker images for particular flows, but this process is detatched. In prefect 1 we had a 1 flow = 1 image policy, but it took a long time to build and development cycles dragged out. It seems the images<->flows connection is a bit more central to you, so if you do need a particular block to correspond to a particular version of a flow I would look at using a randomly generated string