Hi folks! Would like to pick your brains regarding the CICD side of Prefect2. We currently are running flows via ECS with S3 storage. We have multiple different flows that don't share the same dependencies. It seems inefficient and error prone to build one large image for all of these flows to run in. So my question is: • Is anyone using a mono repo for prefect2 with multiple flows each having their own image and requirements files getting deployed via github actions or any other automated deployment tool? • If so, any insight into how you are building images with different requirements files in an automated fashion would be really helpful. Thank you!

You can create different deployments for different flows that use different images

@Khuyen Tran We have set that up manually right now, but looking to see if anyone has automated this process, so that when a PR to main is completed, an image rebuilds with the correct requirements.txt file for that specific image

So the requirements of the images of your flows change frequently?

The requirements are different for each flow. So if we have a few flows that deal with dbt and a few that deal with machine learning, the requirements.txt files will be much different. It'd be great to automate the image build process for each of the flows with separate requirements for each

so I guess right now you are building the docker image for each flow?

correct doing it manually

Hey Zach! My team are using the same multi-deployment multi-image setup that you have outlined above. We are using GitHub Actions, a workflow that builds the parent image (with common dependencies e.g. pandas, numpy, prefect) and then triggers another workflow that builds our child images for each execution environment (e.g. classification, regression, etl).

In case any of these recipes are helpful

Thanks so much for the help everybody! @Chris L. appreciate the tips and I'll be sure to use the action mentioned above

We use environment.yml files and use the micromamba GH action https://github.com/marketplace/actions/provision-with-micromamba with caching. Blazing fast dependency resolution and installs. In our experience, the biggest blocker in iteration speed is managing and waiting for different environments to build….sometimes building the env takes longer than the tests to run. It’s really a speed game. And with Prefect 2.0 (or any python micro service setup really), the three key ingredients for a fast CICD workflow imo are: 1. Bundle flow scripts into your image; 2. Build images from cache (but don’t forget to always invalidate the COPY step for your flow scripts; 3. Micromamba

@Chris L. are you building a new container for each flow in this workflow? We are also running on kubernetes (just getting started), and can run flows on the agent, but I was interested in not having to install all the python libraries for every flow in the agent container