Looking for some advice on deploying flows on Amazon ECS see

Question

Looking for some advice on deploying flows on Amazon ECS see thread

Matt Klein · Answer

We run our Prefect flows on Amazon ECS using the Prefect agent Each time we deploy a new version of the code that underlies our flows we create a new Prefect deployment for each our flows which references 1 An infrastructure block that represents ECS `prefect aws ecs ECSTask` 2 A storage block that represents S3 `prefect filesystems S3` So when we create each deployment the deploy process part of our CI CD pipeline copies the entire codebase into the S3 location When the flow is run the execution infrastructure ECS task downloads the codebase out of S3 and runs the downloaded code This setup works but it s inefficient in that the infrastructure ECS task that s running the flow is itself a Docker image that already has the latest flow code contained in it So it s unnecessary to copy that code into S3 at deployment time and to copy that code out of S3 at flow run time Our codebase supporting flows is fairly large so having to push this into and out of S3 significantly slows down both our CI CD pipeline and flow launching But as far as we can tell there s no other workable alternative for running a flow on ECS Are we missing something Is there some way to deploy a flow without specifying a storage block Or by somehow specifying a storage block that represents something like the flow s code can be found on the local filesystem of the infrastructure that s running the flow I see that a recent release is replacing the Prefect agent architecture with work pool and workers if we made that shift would we be able to do away with these inefficiencies

Ivan · Answer

Maybe skip upload will help in this case

Matt Klein · Answer

Thanks for the thought < Ivan> unfortunately I do not believe it does though That will prevent the code from being uploaded at deployment but when the infrastructure goes to run the flow it still expects to find it at the configured storage location There s no way AFAICT to tell the storage location to use itself instead of S3

Nate · Answer

hi < Matt Klein> gt I see that a recent release is replacing the Prefect agent architecture with work pool and workers yep here s a with workers `prefect yaml` it sounds like you d want to do `prefect init` and select the docker recipe and then specify a docker `build` and `push` step to build and push your image in your prefect yaml by specifying a dockerfile yourself in the build step or letting prefect bake your flow code into the image with `dockerfile auto` alternatively just for context you could not bake your code into the image build and push a base image independently and then each deployment could `push` and `pull` from s3 with the steps which sounds more like the status quo

Matt Klein · Answer

Thanks < Nate> I ll look into the new way appreciate the help