If a flow deployment is configured to use a docker image and

Question

If a flow deployment is configured to use a docker image and the prefect agent is running in an ECS task does each flow run within that same ECS task infrastructure Is there a way to get the agent to spin up a new ECS task for each flow run

Austin Weisgrau · Answer

I think this could be done by using `prefect aws ecs ECSTask` in lieu of `prefect task` as a task decorator which will spin up an ECS task for each prefect task to run

Austin Weisgrau · Answer

Anyways I m finding this tutorial to be quite instructive <https towardsdatascience com prefect aws ecs fargate github actions make serverless dataflows as easy as py f6025335effc 8d94> Working my way through it

Austin Weisgrau · Answer

Got the stack working This tutorial is good It s got some gaps in explanation and instruction but they re figure out able

Marcos · Answer

Apologies for resurrecting this old thread I m trying to achieve a similar thing however I m not sure how you can decorate a task with the `prefect aws ECSTask` so that whenever the task gets invoked it runs in it standalone infra bucket

Marcos · Answer

Does the ECS task need to be registered in the Prefect cloud as an infra block beforehand < Austin Weisgrau>

Austin Weisgrau · Answer

I haven t looked into using the task decorator I ve just been creating deployments that run in ecs at the flow level so I m not sure

Marcos · Answer

Right I was intrigued to see if you could literally decorate a Python definition and let it fly in its own independent infra block I m leveraging `prefect deployments run deployment` to trigger heavier jobs as deployments in their own infra blocks

Marcos · Answer

It s a little bit cumbersome because all your deployments must exist beforehand That means having to re deploy upload your source code to the storage layer every time there s a change

Austin Weisgrau · Answer

That s right

Austin Weisgrau · Answer

The dataflowsops tutorial there has a GitHub actions CD script to handle that automatically when changes to flows are pushed to the main branch

Marcos · Answer

Yup That was a great resource to get more clarity on this approach I guess it can be easily scripted in any general purpose CI system I m just migrating from Prefect v1 0 0 so it s taking me some time

Marcos · Answer

Thanks for your reply < Austin Weisgrau>

Austin Weisgrau · Answer

For sure let me know what you end up working out I m curious