<@ULVA73B9P> how can i configure to run flow runs ...
# ask-marvin
a
@Marvin how can i configure to run flow runs simultaneously,do I need to set any concurrency limit
m
thought for 392.9 seconds
Great question! Short answer: you don’t need to set a concurrency limit just to run flow runs simultaneously. In Prefect 3.x, flows will run in parallel as long as your workers and work pool allow it. Here’s how concurrency works and what you can configure: - Workers - A single worker can run multiple flow runs concurrently. You can cap how many with the worker’s limit. - CLI (per-worker cap):
Copy code
prefect worker start --pool "my-pool" --limit 5
- Python (equivalent):
Copy code
from prefect.workers.process import ProcessWorker

    ProcessWorker(work_pool_name="my-pool", limit=5).start()
- If you don’t set a limit, the worker won’t enforce a cap itself; your machine resources and any pool/global limits will be the bottleneck. - Docs: Workers concept - Work pool concurrency - Caps the total number of running flow runs across all workers in the pool. - If not set, there’s no pool-level cap. - Set or clear:
Copy code
prefect work-pool set-concurrency-limit "my-pool" 10
    prefect work-pool clear-concurrency-limit "my-pool"
- Docs: Work pools concept and work-pool CLI - Task-level (tag-based) concurrency limits - Optional throttle on tasks (or flows if you apply the tag to flows) by tag. Only needed if you want to restrict specific workloads. - Create:
Copy code
prefect concurrency-limit create my-tag 3
- Docs: Global/Tag concurrency limits Putting it together - To run flows simultaneously: - Start at least one worker for your work pool. If you want parallelism on a single worker, give it a higher
--limit
(e.g., 5). - Make sure your work pool concurrency limit is not set too low (or unset if you don’t need a cap). - Only use tag-based concurrency limits if you want to intentionally throttle certain runs. Questions to tailor the advice: - Which worker type are you using (e.g., Process, Docker, Kubernetes)? - Are runs currently queuing, or are you just planning capacity? - Do you want a strict cap on parallelism, or do you just want “as many as resources allow”? Helpful references: - Workers: Workers - Work pools and their concurrency limit: Work pools - Global/Tag concurrency: Global concurrency limits