https://prefect.io logo
m

Matt Klein

09/08/2023, 2:55 PM
Looking for some advice on deploying flows on Amazon ECS — see🧵
We run our Prefect flows on Amazon ECS using the Prefect agent. Each time we deploy a new version of the code that underlies our flows, we create a new Prefect deployment for each our flows which references: 1) An infrastructure block that represents ECS (
prefect_aws.ecs.ECSTask
) 2) A storage block that represents S3 (
prefect.filesystems.S3
) So when we create each deployment, the deploy process (part of our CI/CD pipeline) copies the entire codebase into the S3 location. When the flow is run, the execution infrastructure (ECS task) downloads the codebase out of S3 and runs the downloaded code. This setup works, but it’s inefficient in that the infrastructure (ECS task) that’s running the flow is itself a Docker image that already has the latest flow code contained in it. So it’s unnecessary to copy that code into S3 at deployment time, and to copy that code out of S3 at flow run time. Our codebase supporting flows is fairly large, so having to push this into and out of S3 significantly slows down both our CI/CD pipeline and flow launching. But as far as we can tell there’s no other workable alternative for running a flow on ECS. Are we missing something? Is there some way to deploy a flow without specifying a storage block? Or by somehow specifying a storage block that represents something like “the flow’s code can be found on the local filesystem of the infrastructure that’s running the flow”. I see that a recent release is replacing the Prefect agent architecture with work pool and workers -- if we made that shift would we be able to do away with these inefficiencies?
i

Ivan

09/08/2023, 2:58 PM
Maybe --skip-upload will help in this case?
m

Matt Klein

09/08/2023, 3:08 PM
Thanks for the thought @Ivan, unfortunately I do not believe it does though. That will prevent the code from being uploaded at deployment, but when the infrastructure goes to run the flow it still expects to find it at the configured storage location. There’s no way AFAICT to tell the storage location to use “itself” instead of S3.
n

Nate

09/08/2023, 3:37 PM
hi @Matt Klein
I see that a recent release is replacing the Prefect agent architecture with work pool and workers
yep, here's a guide on moving to workers from agents with workers /
prefect.yaml
, it sounds like you'd want to do
prefect init
and select the docker recipe and then specify a docker
build
and
push
step (to build and push your image) in your prefect yaml by specifying a dockerfile yourself in the build step or letting prefect bake your flow code into the image with
dockerfile: auto
alternatively (just for context), you could not bake your code into the image, build and push a base image independently, and then each deployment could
push
and
pull
from s3 with the steps (which sounds more like the status quo)
👍 2
m

Matt Klein

09/09/2023, 2:11 AM
Thanks @Nate — I’ll look into the “new way” — appreciate the help