Hi Team, Question related to Deployments. I have ...
# ask-community
r
Hi Team, Question related to Deployments. I have multiple deployments for various flows defined in “prefect.yaml”. I am using Farget ECS container to run the flow, flow gets built in docker with “`prefect_docker.deployments.steps"` in prefect.yaml All these 3 flows located in the same directory. Part of the deployments then I run below command (as an example). prefect --no-prompt deploy --name flow-a/dev prefect --no-prompt deploy --name flow-b/dev prefect --no-prompt deploy --name flow-a/test each prefect deploy command runs indedpenntly and create docker image. Because the flow are in located in same location in code, docker image creation for 3 times feels redundant. Is there a way It does not have to build docker image 3 times if I want to deploy 3 flows? Or I should be using different ways to deploy?
m
What if you do
prefect --no-prompt deploy --all
?
1
k
if you have a global build step in a
prefect.yaml
with multiple deployments, it'll still try to build the image every time, but it should be cached after the first deployment, so the other deployments should complete pretty quickly
m
@Kevin Grismore @Mattijs De Paepe, yes the deployment completes quickly if the layer already exists in the previous image but in ECR/ docker registry multiples images are created based on the number of deployments. The last deployment image is marked as the latest tag, the other images are redundant and consuming space. Any possibility to avoid this ?
m
@Manoj Ravi I have a global build and push step. The second build completes quickly because the layer already exists and the second push step also completes quickly because the layer already exists in the registry. I use ECR and when you try to push the exact same image it’ll just come up as ‘layer(s) already exists’ so it doesn’t consume space/other images aren’t duplicated.
m
@Mattijs De Paepe, I'm having the same setup. A global build and push step, and 2 deployments. When i used to deploy locally it used to create one, but im using github actions for deployment now, and 2 images are created for every deployment even while i was running prefect deploy --all. Attaching the image for your reference.
r
Thanks guys, (Sad but) Good to see at least it was not only us who had this issues @Kevin Grismore I have build steps separate in each deployment mainly because I had 1 prefect yaml which takes care of deploying to prod and dev both environment so prod flow is flow-a/prod and dev flow is flow-a/dev But then I am thinking should have 2 different prefect.yaml file, is it possible? With Prefect 1 I had this Docker Build function done manually where I had control to only build and push docker image once and then use prefect register command for each flow which just does a job of registering
m
@ravi I woudn’t go with multiple prefect.yaml files. I think either you have the same docker image for your flows in which case use a global build/push step and caching should take care of the rest. Or they’re different in which case you’ll have to (at least partially) build a new image every time, no?
@Manoj Ravi that’s weird. Can you run deploy --all again and share the logs/see what it does?
m
@ravi, I think we are kind of in the same place, for deployments across environment how are you changing the image name ? Could you parameterize it ?
@Mattijs De Paepe, Sure will run and send you the logs in a while.