Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello everyone, is it correct to run 1 agent for each flow?

Hi. I'm using one agent for multiple flows. However, when the flows run simultaneously, each's runtime is longer.
Let's see what feedback others have.

AFAIK and I've been running Prefect for a few months now, you can just have 1 agent for many flows

If you're running flows locally, then yes, this will inevitably slow up execution times, but if you're executing flows via something like Dask then your agent is only involved in starting and communicating with the jobs

I should have mentioned - yes, I've been testing using LocalAgent

For context, we use ECS on AWS or K8s on Azure, our Agent is either a pod or a ECS task, which will both in turn then spin out containers to manage the Prefect flows and the tasks they spin out