Hello Prefect community! I’m trying to figure out if Prefect is the correct service for my use-case and would like to have some input. My currect setup is using a AWS ECS cluster. I want to have serverless setup of flow runs. This means that I want to be able to schedule a flow to run but when the work is done close down the ecs task running on the cluster as these workflows will only run once a day and will run for less than an hour. Correct if my assumptions are wrong. The way that prefect works is be creating an agent running on ECS and listening on the work queue for a flow to be executed. As I want to create a serverless approach of this my solution would be to create a single ECS agent listening for work and the workflows defines for this ECS task is to invoke the boto3 API and run a different ECS task, which executes my specified workflow. When the executing is done close the ECS task. This for me seems like a workaround. Is there a smarter way to go about this setup?

Our ECS task infrastructure block is made for your use case!

Hey Taylor 🙂 Not quite what i’m looking for. This assumes that I want to use github actions to deploy cloudformation. I do not want to do that. I want is to use terraform for infrastructure to setup my cluster/ecs tasks/ecr and what else I might need. I then want to use the prefect cloud and schedule some workflows and have the options to start a workflow from prefect cloud. But from what I can tell I might need to do a mix of terraform and github actions.

there is cloud formation template there in the repo - you can just use it and deploy from CLI if you prefer that over this nice once click setup I prepared to make your life easier 🙂

aws cloudformation deploy \
  --stack-name "xxxxx" \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=CLUSTER_NAME"

Also, if you use the

ECSTask

infrastructure block you won't need to call the boto3 API manually. The block will automatically start an ECS task, run your flow in it, and shut down the task when the flow run completes. Does that cover the part of the process you described that felt like a workaround?

@Anna Geller Not quite what I'm looking for. I have an existing ECS cluster with everything I need. I have the sample workflow in a docker container with Prefect. Now I do not want to have my Prefect agent running 24/7 in the cluster waiting for workflows to be executed because it does not make sense to keep it running 24/7. As this would just be wasted compute. So what I want is to schedule Prefect to run a command like the ECSTask @Ryan Peden mentioned and start the my workflow. But from what I understand I can't do this out of the box. From my understanding of the documentation and the examples I have seen. If we for example look at this recipe. https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent Note that flows will run inside the agent ECS task, as opposed to becoming their own ECS tasks. This is exactly what I want. I want my workflows to become their own ECS tasks and shutdown afterwards. The solutions I currently see are as follows. Solution 1. Have a single Prefect agent running in my ECS cluster and wait for work. The workflows given to this agent is a workflow, which runs the ECSTask command to spin up a second workflow that I actually desire to run. This seems like a work around because for every workflow I create I will need two workflows as the first one is responsible for starting a second ECS task that performs my actual task. Solution 2. As linked earlier. Schedule everything from github actions to start my workflows. This also seems like a work around because then the scheduling becomes the responsibility of Github actions and not Prefect and I can't start my workflows from Prefect.

Thanks for the explanation, Ditlev. Instead of running

ECSTask

manually, you can set it up as an infrastructure block. Then you can use it with your deployments and the agent will use it behind the scenes to run your flows and it should behave exactly the way you want it to. You can create en ECSTask block via the UI, or you can use a Python script like this one. This creates a block containing all the information the agent needs to run your flow as an ECS task and then shut the task down when the flow completes. You'll never need to instantiate

ECSTask

yourself; the agent will do all of that for you. Although you don't want to use GitHub actions, the article linked above still contains a useful example if you skip to the Deploy flows using ECSTask infrastructure blocks heading. So the steps to follow would be: 1. Create an

ECSTask

infrastructure block via the Prefect UI or in a Python script like the one linked above. 2. Use the infrastructure block in your deployment. This can be as simple as adding

-ib ecs-task/prod

to the deployment CLI command (note that

prod

is just an example; you can use any block name you want), or if you are creating your deployment in Python it would look something like:

from prefect.deployments import Deployment
from prefect_aws.ecs import ECSTask

ecs_infrastructure = ECSTask.load("prod")

my_deployment = Deployment.build_from_flow(
    flow=my_flow_function,
    name="Test Flow",
    infrastructure=ecs_infrastructure, 
    schedule=<your schedule here>,
    work_queue="my_work_queue"
)

my_deployment.apply()

3. Run an agent on ECS that watches your deployment's work queue. And that's it; when the agent picks up you scheduled flows, it will run each flow in a separate ECS task.

@Ryan Peden Thanks for the great explanation! I believe this was exactly what I needed. And thanks to all of you guys for the fast response. I really appreciate it.

You're welcome! I'm happy it was useful. Thanks for being part of the Prefect community! 😄