Hey team,
While executing flows using subprocess o...
# ask-communitya
Aditya N
05/04/2023, 11:25 AMHey team,
While executing flows using subprocess on an agent/worker (and ConcurrentTaskRunner), what's the max no of concurrent flows I can run ? From what I understand, each sub-process can only run one flow at a time which means the event-loop (of that process) is shared only by tasks of that flow. So assuming the agent is running on a 8vCPU machine, can I only have about 16 flow runs in parallel ?
z
Zanie
05/04/2023, 3:19 PM
The agent will spawn a unique process per flow run, yes.
Zanie
05/04/2023, 3:20 PM
You can generally exceed far more than your number of cores in number of flow runs, i.e. you have far more processes running on your machine than you do cores. However, if your flows are computationally intensive you will encounter slowdowns eventually as context switching adds overhead.
a
Aditya N
06/03/2023, 1:33 PMHey @Zanie, are there any caveats to watch out for if process type worker is deployed in production ? Coz the docs page says this it typically used for getting started / testing purposes. Thanks!
z
Zanie
06/03/2023, 5:06 PM
Nope lots of people use it in production!
Zanie
06/03/2023, 5:07 PM
You may want to set some concurrency limits since the flow runs execute locally instead of being managed by something like Kubernetes