Ditlev Stjerne
11/28/2022, 11:13 AMTaylor Curran
11/28/2022, 2:34 PMDitlev Stjerne
11/28/2022, 2:55 PMAnna Geller
11/28/2022, 6:04 PMaws cloudformation deploy \
--stack-name "xxxxx" \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides "ClusterName=CLUSTER_NAME"
Ryan Peden
11/28/2022, 6:09 PMECSTask
infrastructure block you won't need to call the boto3 API manually. The block will automatically start an ECS task, run your flow in it, and shut down the task when the flow run completes. Does that cover the part of the process you described that felt like a workaround?Ditlev Stjerne
11/28/2022, 6:48 PMRyan Peden
11/28/2022, 7:37 PMECSTask
manually, you can set it up as an infrastructure block. Then you can use it with your deployments and the agent will use it behind the scenes to run your flows and it should behave exactly the way you want it to.
You can create en ECSTask block via the UI, or you can use a Python script like this one. This creates a block containing all the information the agent needs to run your flow as an ECS task and then shut the task down when the flow completes. You'll never need to instantiate ECSTask
yourself; the agent will do all of that for you.
Although you don't want to use GitHub actions, the article linked above still contains a useful example if you skip to the Deploy flows using ECSTask infrastructure blocks heading.
So the steps to follow would be:
1. Create an ECSTask
infrastructure block via the Prefect UI or in a Python script like the one linked above.
2. Use the infrastructure block in your deployment. This can be as simple as adding -ib ecs-task/prod
to the deployment CLI command (note that prod
is just an example; you can use any block name you want), or if you are creating your deployment in Python it would look something like:
from prefect.deployments import Deployment
from prefect_aws.ecs import ECSTask
ecs_infrastructure = ECSTask.load("prod")
my_deployment = Deployment.build_from_flow(
flow=my_flow_function,
name="Test Flow",
infrastructure=ecs_infrastructure,
schedule=<your schedule here>,
work_queue="my_work_queue"
)
my_deployment.apply()
3. Run an agent on ECS that watches your deployment's work queue.
And that's it; when the agent picks up you scheduled flows, it will run each flow in a separate ECS task.Ditlev Stjerne
11/28/2022, 8:00 PMRyan Peden
11/28/2022, 8:05 PM