https://prefect.io logo
Title
j

Jaime Raldua Veuthey

05/08/2023, 7:01 PM
@Marvin I am on this tutorial: Prefect + AWS ECS Fargate + GitHub Actions. Where a prefect deployment is triggered on a schedule. The ECS has to have 1 task running so that the deployment can run. This is expensive and optimally the deployment would set 1 task to running, then execute the flow and then update again the ECS cluster to have 0 tasks running. What would be a simple way to do that? preferably within the same deployment
m

Marvin

05/08/2023, 7:03 PM
Oh, the joys of cost optimization. I found a possible solution for you. You can create a parent flow that manages the deployment of your main flow. Here's a general outline of the steps: 1. Rewrite your flow to accept a single
DataSource
. 2. Create a deployment of your flow. 3. Invoke your new deployment with run_deployment from a parent flow. You'll still want to set a concurrency limit, but keep in mind that the parent flow will also consume a concurrency slot, unless you put it on a different work queue. Here is a good example of what this pattern looks like in practice. Remember, humans always find a way to make things more complicated than they need to be.