Ditlev Stjerne11/28/2022, 11:13 AM
Taylor Curran11/28/2022, 2:34 PM
Ditlev Stjerne11/28/2022, 2:55 PM
Anna Geller11/28/2022, 6:04 PM
aws cloudformation deploy \ --stack-name "xxxxx" \ --capabilities CAPABILITY_NAMED_IAM \ --parameter-overrides "ClusterName=CLUSTER_NAME"
Ryan Peden11/28/2022, 6:09 PM
infrastructure block you won't need to call the boto3 API manually. The block will automatically start an ECS task, run your flow in it, and shut down the task when the flow run completes. Does that cover the part of the process you described that felt like a workaround?
Ditlev Stjerne11/28/2022, 6:48 PM
Ryan Peden11/28/2022, 7:37 PM
manually, you can set it up as an infrastructure block. Then you can use it with your deployments and the agent will use it behind the scenes to run your flows and it should behave exactly the way you want it to. You can create en ECSTask block via the UI, or you can use a Python script like this one. This creates a block containing all the information the agent needs to run your flow as an ECS task and then shut the task down when the flow completes. You'll never need to instantiate
yourself; the agent will do all of that for you. Although you don't want to use GitHub actions, the article linked above still contains a useful example if you skip to the Deploy flows using ECSTask infrastructure blocks heading. So the steps to follow would be: 1. Create an
infrastructure block via the Prefect UI or in a Python script like the one linked above. 2. Use the infrastructure block in your deployment. This can be as simple as adding
to the deployment CLI command (note that
is just an example; you can use any block name you want), or if you are creating your deployment in Python it would look something like:
3. Run an agent on ECS that watches your deployment's work queue. And that's it; when the agent picks up you scheduled flows, it will run each flow in a separate ECS task.
from prefect.deployments import Deployment from prefect_aws.ecs import ECSTask ecs_infrastructure = ECSTask.load("prod") my_deployment = Deployment.build_from_flow( flow=my_flow_function, name="Test Flow", infrastructure=ecs_infrastructure, schedule=<your schedule here>, work_queue="my_work_queue" ) my_deployment.apply()
Ditlev Stjerne11/28/2022, 8:00 PM
Ryan Peden11/28/2022, 8:05 PM