https://prefect.io logo
Title
a

Austin Weisgrau

02/09/2023, 10:51 PM
If a flow deployment is configured to use a docker image, and the prefect agent is running in an ECS task, does each flow run within that same ECS task infrastructure? Is there a way to get the agent to spin up a new ECS task for each flow run?
I think this could be done by using
prefect_aws.ecs.ECSTask
in lieu of
prefect.task
as a task decorator, which will spin up an ECS task for each prefect task to run
Anyways I'm finding this tutorial to be quite instructive: https://towardsdatascience.com/prefect-aws-ecs-fargate-github-actions-make-serverless-dataflows-as-easy-as-py-f6025335effc#8d94 Working my way through it
👀 1
Got the stack working! This tutorial is good. It's got some gaps in explanation and instruction but they're figure-out-able
m

Marcos

03/10/2023, 3:48 PM
Apologies for resurrecting this old thread. I'm trying to achieve a similar thing, however, I'm not sure how you can decorate a task with the
prefect_aws.ECSTask
so that whenever the task gets invoked, it runs in it standalone infra bucket.
Does the ECS task need to be registered in the Prefect cloud as an infra block beforehand? @Austin Weisgrau
a

Austin Weisgrau

03/10/2023, 3:57 PM
I haven't looked into using the task decorator, I've just been creating deployments that run in ecs at the flow level, so I'm not sure
👍 1
m

Marcos

03/10/2023, 3:59 PM
Right. I was intrigued to see if you could literally decorate a Python definition and let it fly in its own independent infra block. I'm leveraging
prefect.deployments.run_deployment
to trigger heavier jobs as deployments in their own infra blocks.
It's a little bit cumbersome because all your deployments must exist beforehand. That means having to re-deploy/upload your source code to the storage layer every time there's a change.
a

Austin Weisgrau

03/10/2023, 4:00 PM
That's right
The dataflowsops tutorial there has a GitHub actions CD script to handle that automatically when changes to flows are pushed to the main branch
m

Marcos

03/10/2023, 4:02 PM
Yup. That was a great resource to get more clarity on this approach. I guess it can be easily scripted in any general-purpose CI system. I'm just migrating from Prefect v1.0.0 so it's taking me some time!
👍 1
Thanks for your reply @Austin Weisgrau!
a

Austin Weisgrau

03/10/2023, 4:56 PM
For sure, let me know what you end up working out! I'm curious
👍 1