Hi, we are evaluating using prefect to run our ML pipelines. Currently these are run as a series of ECS tasks on Fargate, triggered periodically through Cloudwatch. The big contender for the job is Airflow, here our setup would look something like this: have Airflow UI and scheduler running in the cloud (probably also as ECS tasks), and have a DAG which triggers the existing pieces of the pipeline in ECS. So far I haven't been able to find a similar functionality for prefect. Here's what I understood: we would run the prefect UI and server in the cloud, and have an ECS agent deployed that talks to the server. This agent would be able to run entire workflows within a dedicated ECS task, but I would rather trigger existing tasks. Would love if you could point me to documentation that explains the architecture, and maybe covers our use case as well. Thanks!

Another community member posted this recently: https://lejimmy.com/distributed-data-pipelines-with-aws-ecs-fargate-and-prefect-cloud/

Because Prefect exposes first-class dataflow, running tasks in isolation becomes tricky because we need a way of sending a task the necessary inputs. Our current recommendation for this sort of pattern is that any piece of a DAG that you want to be able to run in isolation should be its own flow, and you can wire your flows together into a larger meta-graph using https://docs.prefect.io/core/idioms/flow-to-flow.html

Right, thanks very much. It's a different model than the one Airflow uses, more integrated into the code. So in order to use prefect, we would rewrite our pipeline into proper prefect workflows which would then be picked up by the ECS agents, right? If I did want to trigger a singular ECS task from the workflow however, how would I go about that? Airflow has the ECSOperator, would I just use boto3 directly?

Yea at this moment we don’t have an off-the-shelf

ECSTask

so you’d need to use

boto3

directly; however, if you got something working and would be interested in making a pull request, we’d love to get that merged into our task library so other users can take advantage! For reference, here are some other AWS integrations we already have: https://docs.prefect.io/api/latest/tasks/aws.html

Awesome, if we decide to go that route, I will make sure to contribute the code we write!