https://prefect.io logo
Title
n

nib

12/22/2020, 10:17 AM
Hi, a question about agent. I created 1k flow runs. LocalAgent started to deploy all these flows using popen(prefect execute flow-run). And it created 1k processes. All this processes used all available memory. Is it possible to limit number of deployed flows? I use OSS version. Thanks in advance
👀 2
j

Jim Crist-Harif

12/22/2020, 3:54 PM
Not for the local agent. Agents have no knowledge of the backing infrastructure's resource usage, relying instead on the backing platform to have some job queuing mechanism. For example, the k8s agent creates kubernetes jobs - creating 1000 kubernetes jobs won't immediately start 1000 kubernetess jobs, kubernetes will queue the jobs running them within the cluster resource limits. Local processes have no such queuing, users are currently responsible for scheduling jobs in such a way as to not overcommit their resources.
Note that Prefect Cloud does have some concurrency limiting features that don't exist in Prefect Server.
g

Giovanni Giacco

12/22/2020, 6:20 PM
I guess a local Dask Executor should queue the requests... am I right?
j

Jim Crist-Harif

12/22/2020, 6:22 PM
Executors are for running multiple task runs - the problem above has to do with multiple flow runs. These are unrelated.