If a flow deployment is configured to use a docker...
# prefect-getting-starteda
Austin Weisgrau
02/09/2023, 10:51 PMIf a flow deployment is configured to use a docker image, and the prefect agent is running in an ECS task, does each flow run within that same ECS task infrastructure? Is there a way to get the agent to spin up a new ECS task for each flow run?
Austin Weisgrau
02/09/2023, 10:56 PMI think this could be done by using
prefect_aws.ecs.ECSTaskprefect.taskAustin Weisgrau
02/10/2023, 1:11 AMAnyways I'm finding this tutorial to be quite instructive: https://towardsdatascience.com/prefect-aws-ecs-fargate-github-actions-make-serverless-dataflows-as-easy-as-py-f6025335effc#8d94
Working my way through it
👀 1
Austin Weisgrau
02/10/2023, 2:52 AMGot the stack working! This tutorial is good. It's got some gaps in explanation and instruction but they're figure-out-able
m
Marcos
03/10/2023, 3:48 PMApologies for resurrecting this old thread. I'm trying to achieve a similar thing, however, I'm not sure how you can decorate a task with the
prefect_aws.ECSTaskMarcos
03/10/2023, 3:48 PMDoes the ECS task need to be registered in the Prefect cloud as an infra block beforehand? @Austin Weisgrau
a
Austin Weisgrau
03/10/2023, 3:57 PMI haven't looked into using the task decorator, I've just been creating deployments that run in ecs at the flow level, so I'm not sure
👍 1
m
Marcos
03/10/2023, 3:59 PMRight. I was intrigued to see if you could literally decorate a Python definition and let it fly in its own independent infra block. I'm leveraging
prefect.deployments.run_deploymentMarcos
03/10/2023, 4:00 PMIt's a little bit cumbersome because all your deployments must exist beforehand. That means having to re-deploy/upload your source code to the storage layer every time there's a change.
a
Austin Weisgrau
03/10/2023, 4:00 PMThat's right
Austin Weisgrau
03/10/2023, 4:01 PMThe dataflowsops tutorial there has a GitHub actions CD script to handle that automatically when changes to flows are pushed to the main branch
m
Marcos
03/10/2023, 4:02 PMYup. That was a great resource to get more clarity on this approach. I guess it can be easily scripted in any general-purpose CI system. I'm just migrating from Prefect v1.0.0 so it's taking me some time!
👍 1
Marcos
03/10/2023, 4:05 PMThanks for your reply @Austin Weisgrau!
a
Austin Weisgrau
03/10/2023, 4:56 PMFor sure, let me know what you end up working out! I'm curious
👍 1