Alvaro Durán Tovar

12/11/2021, 6:42 PM
How do you recommend approaching CI pipelines with docker storage? So I think we might end up having N flows in python scripts, using docker as storage deploying to kubernetes. I guess we will run all flows on every release, that means creating a new docker image for each of the flows and registering them again. Can be ok, I mean, I not going to do it manually... but seems a bit overkill to rebuild and upgrade all flows if only 1 has changed. Any recommendation on how to manage flows through CI pipelines if we have all flows on the same repo?

Kevin Kho

12/11/2021, 7:56 PM
Hi @Alvaro Durán Tovar, have you seen this page?
This discussion thread might also have some info about this

Alvaro Durán Tovar

12/11/2021, 8:25 PM
nice discussion, thanks!

Billy McMonagle

12/12/2021, 2:53 AM
Hi Alvaro, my post is one of those in the discussion Kevin posted, but I can elaborate slightly bc we're running in production now... I am using custom docker images for each of my flow groups. For performance, I rely heavily on some of the caching features available in buildkit, and build my flows with a makefile that runs multiple builds in parallel. Right now I have 1 base image and then 10 different flow images that extend it. Typical build time is around 4 minutes. Like you, I'm also interested in ways to selectively rebuild individual images, but I am hoping to put off fully optimizing this for a while... by making effective use of build caching, it really doesn't matter if everything is getting rebuilt.

Anna Geller

12/12/2021, 2:12 PM
@Alvaro Durán Tovar I think the problem is not specific to Prefect, but in general building CI/CD pipelines in monorepos is a bit more challenging. You could have a look at this blog post from AWS showing how you could e.g. use GitHub event payload to determine what a CI pipeline should do e.g. triggering only build for Project A and building a Docker image only for that flow: https://aws.amazon.com/blogs/devops/integrate-github-monorepo-with-aws-codepipeline-to-run-project-specific-ci-cd-pipelines/. Based on this, it seems that many CI tools have option to filter based on path in the repository.