Hi, we're using and evaluating prefect and are loving it. Our question is: How to handle continuous deployments with prefect? We're using gitlab pipelines to deploy code and register prefect flows any time we push in the master branch. Can we tell prefect which python environment and which version of our custom code to use for each flow? So that we don't break running flows during a deployment? Thanks in advance!

Hi! I don’t have personal experience with continuous deployments but I know some folks in the slack definitely do so they may chime in too. But my 2 cents is that you should be using Docker storage to specify any detail about the run environment (python version, your code version, etc) for a flow by providing your own Dockerfile. Registering a flow will trigger a docker build off that Dockerfile, which will be tagged separately (as in, old images from prior versions of flows will still exist). The accompanying agent will then use the target image for a given flow to deploy that flow, so they won’t affect each other. In the case of already running flows, since they are isolated in their own docker image their run environment won’t be affected by a new deploy. Are you already using Docker storage? If not, here is a quick link to the docs: https://docs.prefect.io/orchestration/execution/storage_options.html#docker

Hi @Emmanuel Klinger, I totally agree with what @Laura Lorenz (she/her) described above, i.e. building a docker image is the best way to capture all of the dependencies and code that your Flows will use. Here's some detail on the process that we use for CICD, specifically how we put all of our Flows into one docker image that we build with our CICD pipeline: https://prefect-community.slack.com/archives/CL09KU1K7/p1588598737005100?thread_ts=1588164482.312600&cid=CL09KU1K7

Thanks a lot @Laura Lorenz (she/her) and @Joe Schmid! I'm always amazed by the great community here.

Hey there. I am wondering how the new cloud storage option might change this approach. With the caveat that I am still wrapping my mind around all the pieces, I wonder if you still would want to package the flows in a docker image or if the image would just have the execution environment and the flow code would be stored in cloud storage. If you'd use the latter, how are the flows versioned in cloud storage? Would the build process for the flow code in cloud storage simply be the registration of the flow?

@Pedro Machado My two cents is: cloud storage (S3, GCS, etc.) provides some nice flexibility specifically for storing serialized Flows, but it doesn't address dependencies like python modules, your own Python library code used by Flows, non-python dependencies e.g. database drivers, etc. Docker directly addresses dependencies, i.e. by building a Docker image you know exactly the set of dependencies will be available with the expected versions, etc. so it's a great fit for CICD, just like @Laura Lorenz (she/her) described above.

Yes exactly @Pedro Machado the build process for the flows themselves when used in ‘non-docker storage for containerized environments’ (heh…. we picked a harsh name there) is registration, and they are versioned the same way as GCS / S3 flow storage works when its on its own, which is a slug that includes the build timestamp. If your flows are slow-changing then packaging them up in the docker file using the normal Docker storage type is definitely fine and simpler to reason about / configure, but the “non-docker storage for containerized environments” definitely shines if you have fast changing flows that you are constantly registering (and with pure Docker storage, this means a docker image build every registration).

Makes sense. Thank you Joe and Laura!