https://prefect.io logo
Title
c

Cody

05/09/2023, 4:13 PM
I'm looking to run flows using AWS ECS/ECR. My flow code will live entirely inside the docker images. The plan is for one "initial" flow/container to call another flow/container after completion, and do this three or so times until my ETL process is complete. Does anyone use this sort of setup, and if so, do you use a different ECS Task Definition for each Flow/Container? I'm fairly new to ECS, so I'm not sure if I can use multiple containers under one ECS Task Definition (the TD effectively acting as a way to give each container the credentials it needs to run) choosing just the container I want for a given flow, or if each flow/container needs its own ECS Task Definition. Any help would be greatly appreciated!
j

jack

05/09/2023, 5:00 PM
One pattern we've used is to build a single docker image containing the code for multiple prefect flows. Then create a single ECSTask infrastructure block (which points to the docker image). Each flow gets its own deployment, and each deployment points to the single ECSTask infra block.
c

Cody

05/09/2023, 5:01 PM
That makes a lot of sense. I'll try that. Thanks!
j

jack

05/09/2023, 5:07 PM
Make sure not to specify any
storage
for the deployments. But use the
path
keyword arg when creating the deployment to point to the directory in the docker image where the flow code lives. (And make sure
path
ends in a trailing slash.)