Eric
07/18/2025, 9:59 AMMarvin
07/18/2025, 9:59 AMMarvin
07/18/2025, 10:00 AMbash
prefect work-pool set-concurrency-limit <pool-name> <limit>
2. Worker-Level Limits: When starting a worker, you can set a limit on the number of concurrent flow runs using the --limit
flag:
bash
prefect worker start -p "my-pool" --limit <number>
3. Resource Constraints: The actual number of concurrent flows will also be limited by the available system resources (CPU, memory) of the Docker container and host system.
Important notes:
- If no explicit limits are set, a worker can theoretically handle multiple flows simultaneously
- You should carefully consider resource requirements when setting concurrency limits
- Setting appropriate concurrency limits is important for preventing resource exhaustion
- If you need to handle many flows, you can run multiple workers for the same work pool
Best Practices:
1. Start with conservative concurrency limits and adjust based on your flow's resource requirements
2. Monitor system resources when running concurrent flows
3. Consider using multiple workers if you need to scale horizontally
So to directly answer your question: No, a Docker worker is not limited to handling only one main flow at a time unless you explicitly configure it that way through concurrency limits.