Matt Klein
09/08/2023, 2:55 PMprefect_aws.ecs.ECSTask
)
2) A storage block that represents S3 (prefect.filesystems.S3
)
So when we create each deployment, the deploy process (part of our CI/CD pipeline) copies the entire codebase into the S3 location. When the flow is run, the execution infrastructure (ECS task) downloads the codebase out of S3 and runs the downloaded code.
This setup works, but it’s inefficient in that the infrastructure (ECS task) that’s running the flow is itself a Docker image that already has the latest flow code contained in it. So it’s unnecessary to copy that code into S3 at deployment time, and to copy that code out of S3 at flow run time.
Our codebase supporting flows is fairly large, so having to push this into and out of S3 significantly slows down both our CI/CD pipeline and flow launching. But as far as we can tell there’s no other workable alternative for running a flow on ECS. Are we missing something? Is there some way to deploy a flow without specifying a storage block? Or by somehow specifying a storage block that represents something like “the flow’s code can be found on the local filesystem of the infrastructure that’s running the flow”.
I see that a recent release is replacing the Prefect agent architecture with work pool and workers -- if we made that shift would we be able to do away with these inefficiencies?Ivan
09/08/2023, 2:58 PMMatt Klein
09/08/2023, 3:08 PMNate
09/08/2023, 3:37 PMI see that a recent release is replacing the Prefect agent architecture with work pool and workersyep, here's a guide on moving to workers from agents with workers /
prefect.yaml
, it sounds like you'd want to do
prefect init
and select the docker recipe
and then specify a docker build
and push
step (to build and push your image) in your prefect yaml by specifying a dockerfile yourself in the build step or letting prefect bake your flow code into the image with dockerfile: auto
alternatively (just for context), you could not bake your code into the image, build and push a base image independently, and then each deployment could push
and pull
from s3 with the steps (which sounds more like the status quo)Matt Klein
09/09/2023, 2:11 AM