https://prefect.io
Join Slack
If a flow deployment is configured to use a docker image, and the prefect agent is running in an ECS...
# prefect-getting-started
a
If a flow deployment is configured to use a docker image, and the prefect agent is running in an ECS task, does each flow run within that same ECS task infrastructure? Is there a way to get the agent to spin up a new ECS task for each flow run?
I think this could be done by using 
prefect_aws.ecs.ECSTask
in lieu of 
prefect.task
as a task decorator, which will spin up an ECS task for each prefect task to run
Anyways I'm finding this tutorial to be quite instructive: https://towardsdatascience.com/prefect-aws-ecs-fargate-github-actions-make-serverless-dataflows-as-easy-as-py-f6025335effc#8d94 Working my way through it
👀 1
Got the stack working! This tutorial is good. It's got some gaps in explanation and instruction but they're figure-out-able
m
Apologies for resurrecting this old thread. I'm trying to achieve a similar thing, however, I'm not sure how you can decorate a task with the 
prefect_aws.ECSTask
so that whenever the task gets invoked, it runs in it standalone infra bucket.
Does the ECS task need to be registered in the Prefect cloud as an infra block beforehand? @Austin Weisgrau
a
I haven't looked into using the task decorator, I've just been creating deployments that run in ecs at the flow level, so I'm not sure
👍 1
m
Right. I was intrigued to see if you could literally decorate a Python definition and let it fly in its own independent infra block. I'm leveraging 
prefect.deployments.run_deployment
to trigger heavier jobs as deployments in their own infra blocks.
It's a little bit cumbersome because all your deployments must exist beforehand. That means having to re-deploy/upload your source code to the storage layer every time there's a change.
a
That's right
The dataflowsops tutorial there has a GitHub actions CD script to handle that automatically when changes to flows are pushed to the main branch
m
Yup. That was a great resource to get more clarity on this approach. I guess it can be easily scripted in any general-purpose CI system. I'm just migrating from Prefect v1.0.0 so it's taking me some time!
👍 1
Thanks for your reply @Austin Weisgrau!
a
For sure, let me know what you end up working out! I'm curious
👍 1
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by