How do you recommend approaching CI pipelines with docker storage? So I think we might end up having N flows in python scripts, using docker as storage deploying to kubernetes. I guess we will run all flows on every release, that means creating a new docker image for each of the flows and registering them again. Can be ok, I mean, I not going to do it manually... but seems a bit overkill to rebuild and upgrade all flows if only 1 has changed. Any recommendation on how to manage flows through CI pipelines if we have all flows on the same repo?

Hi @Alvaro Durán Tovar, have you seen this page?

nice discussion, thanks!

Hi Alvaro, my post is one of those in the discussion Kevin posted, but I can elaborate slightly bc we're running in production now... I am using custom docker images for each of my flow groups. For performance, I rely heavily on some of the caching features available in buildkit, and build my flows with a makefile that runs multiple builds in parallel. Right now I have 1 base image and then 10 different flow images that extend it. Typical build time is around 4 minutes. Like you, I'm also interested in ways to selectively rebuild individual images, but I am hoping to put off fully optimizing this for a while... by making effective use of build caching, it really doesn't matter if everything is getting rebuilt.

@Alvaro Durán Tovar I think the problem is not specific to Prefect, but in general building CI/CD pipelines in monorepos is a bit more challenging. You could have a look at this blog post from AWS showing how you could e.g. use GitHub event payload to determine what a CI pipeline should do e.g. triggering only build for Project A and building a Docker image only for that flow: https://aws.amazon.com/blogs/devops/integrate-github-monorepo-with-aws-codepipeline-to-run-project-specific-ci-cd-pipelines/. Based on this, it seems that many CI tools have option to filter based on path in the repository.