Hello everyone, is it correct to run 1 agent for each flow?
s
Stéphan Taljaard
07/05/2021, 12:50 PM
Hi. I'm using one agent for multiple flows. However, when the flows run simultaneously, each's runtime is longer.
Let's see what feedback others have.
c
ciaran
07/05/2021, 1:41 PM
AFAIK and I've been running Prefect for a few months now, you can just have 1 agent for many flows
If you're running flows locally, then yes, this will inevitably slow up execution times, but if you're executing flows via something like Dask then your agent is only involved in starting and communicating with the jobs
s
Stéphan Taljaard
07/05/2021, 1:55 PM
I should have mentioned - yes, I've been testing using LocalAgent
c
ciaran
07/05/2021, 1:58 PM
For context, we use ECS on AWS or K8s on Azure, our Agent is either a pod or a ECS task, which will both in turn then spin out containers to manage the Prefect flows and the tasks they spin out