Great question — and you’re absolutely in the right direction thinking about scaling strategy. Short answer: - Multiple workers is not just okay — it’s the recommended approach at scale. Prefect 2.x workers are designed to run safely in parallel on the same work pool/queue. The API uses row-level locking to ensure a flow run is only claimed once, so you won’t get duplicate execution. - For “thousands of concurrent inference jobs,” favor horizontal scaling (many workers) over vertical scaling (one giant worker). Vertical scaling helps each individual run, but it won’t increase how many runs a single worker can submit/manage at once. Horizontal scaling gives you throughput, resiliency, and elasticity. Recommended architecture for 1,000s of concurrent runs (Prefect 2.x): - Use a per-run ephemeral execution environment (e.g., Kubernetes Job, ECS Task, Docker container) so each inference runs in its own isolated container with the right CPU/GPU/memory. - Run many lightweight workers in parallel pointing at the same work pool/queue(s). - Start with 20–100 workers, each with a limit of 25–100 concurrent submissions, and tune from there. - Set an overall work pool concurrency limit to cap total throughput and prevent overload. - Ensure your orchestration DB is PostgreSQL (SQLite will bottleneck quickly). If you host the Prefect API yourself, use connection pooling (e.g., PgBouncer). Concrete knobs to use: - Worker-level:

--limit

caps how many flow runs a single worker will submit simultaneously. - Work pool-level: a global concurrency limit across all queues in the pool. - Work queue-level: per-queue concurrency limit if you shard by queue. - Tag-based concurrency limits: use tags to cap certain classes of runs (e.g., GPU-bound) if you’re on Prefect Cloud; on OSS Server, emulate this with dedicated queues and queue limits. CLI you’ll actually use (Prefect 2.x): - Create a work pool (example: Kubernetes):

prefect work-pool create "inference-pool" --type kubernetes

- Start multiple workers against the same pool (safe to run many in parallel):

prefect worker start --pool "inference-pool" --limit 50 --name "inference-worker-1"
prefect worker start --pool "inference-pool" --limit 50 --name "inference-worker-2"
# ... scale out to N workers

You can run these as K8s Deployments with N replicas or as separate processes/containers on different hosts. Patterns that work well for inference: - Make each flow run represent a single inference job (or a small batch). Submit many runs in parallel via a deployment. - If you have GPUs, set resource requests/limits in your base job template and use K8s scheduling (node selectors, tolerations) to place runs on GPU nodes. Use tag-based concurrency or separate queues to cap runs per GPU type. - Shard queues by priority, model family, or resource type if it helps control fairness and capacity. Operational caveats to plan for: - Database connections: lots of workers mean lots of connections. Use Postgres + pooling if you self-host. - API and cluster rate limits: for very high churn (thousands of short-lived jobs), watch K8s/ECS API quotas and adjust worker

--limit

and total concurrency to avoid bursts. - Don’t run a single “mega worker” — multiple small/medium workers reduce blast radius and improve responsiveness. - Avoid SQLite in production. It will become a bottleneck well before you reach “thousands.” A baseline to try: - Work pool concurrency limit: 2000 (or your desired cap) - 40 workers, each

--limit 50

(total capacity ~2000 active submissions) - Kubernetes Jobs with appropriate CPU/GPU requests - If on OSS Server: Postgres + PgBouncer Happy to help you size this precisely. A few details that would let me tailor the plan: - Are you on Prefect Cloud or self-hosted Prefect Server? - Which worker type/infrastructure (Kubernetes, ECS, Docker on VMs)? - Do your inference runs need GPUs? If so, how many GPUs per run and what models?

- Typical run duration and the max concurrent runs you want to support at steady state? Once I have that, I can provide a right-sized pool/queue layout, example base job templates, and concrete concurrency settings.

@Marvin where in the helm chart do I define this limit? Also if this identical for prefect 3. I have both versions running in my k8s clusters

Great question. In the Prefect Helm chart, the worker’s concurrency limit is set in values.yaml at worker.config.limit. This is the same for both Prefect 2.x and 3.x in the prefect-worker chart. Where to set it (both 2.x and 3.x): - Chart: prefect-worker - Key: worker.config.limit Example values.yaml:

worker:
  config:
    workPool: "my-pool"
    type: kubernetes        # or docker/process/etc.
    limit: 50               # this becomes `--limit 50`
  replicaCount: 10          # scale horizontally via replicas

Alternative overrides: - Append flags without replacing the whole command:

worker:
  extraArgs:
    - --limit
    - "50"

- Fully override the command/args (rarely needed):

worker:
  args:
    - prefect
    - worker
    - start
    - --type
    - kubernetes
    - --pool
    - my-pool
    - --limit
    - "50"

Pool-level concurrency limit (separate from worker.limit): - This caps flow runs across all workers in a pool and is not set via Helm. Use the CLI:

prefect work-pool set-concurrency-limit "my-pool" 500
prefect work-pool clear-concurrency-limit "my-pool"

Links: - Prefect Helm charts: prefect-helm repo - Helm chart docs index: prefect-helm docs - CLI reference checked:

prefect worker start --help

,

prefect work-pool set-concurrency-limit --help

If you’re running a custom Deployment (not the Helm chart), just add

--limit <N>

to the container args:

args:
  - prefect
  - worker
  - start
  - --type
  - kubernetes
  - --pool
  - my-pool
  - --limit
  - "50"

Want me to sketch a full values.yaml for your setup (2.x and 3.x) based on your worker type(s) and target concurrency?

Thanks for calling that out — a few clarifications: - HPA is not enabled by default in the prefect-worker Helm chart. By default: - worker.autoscaling.enabled: false - worker.autoscaling.minReplicas: 1 - worker.autoscaling.maxReplicas: 1 - worker.replicaCount: 1 - The per-worker concurrency limit is also not set by default: - worker.config.limit: null (meaning no explicit cap unless you set it) So the default chart does not autoscale out-of-the-box and does not cap per-worker concurrency — it’s a conservative, single-replica baseline. There’s no anti-pattern “by default.” The anti-pattern would be enabling autoscaling without also putting concurrency controls in place. Recommended pattern when you enable autoscaling: - Always set a per-worker limit (worker.config.limit) to prevent a single pod from submitting too many runs at once. - Optionally, set a work pool concurrency limit to cap total throughput across all workers in the pool. Example values.yaml (Prefect 3.x Helm chart):

worker:
  config:
    workPool: "inference-pool"
    type: kubernetes
    limit: 50                 # per-worker cap -> --limit 50
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 40
    targetCPUUtilizationPercentage: 70

Then cap pool-wide throughput (Prefect 2.x and 3.x):

prefect work-pool set-concurrency-limit "inference-pool" 1000

Notes for your mixed 2.x / 3.x environment: - The Helm chart in prefect-helm targets Prefect 3.x. If you’re running Prefect 2.x workers, you’ll typically set

--limit

via your own Deployment args (or ensure your chart passes that flag):

args:
  - prefect
  - worker
  - start
  - --type
  - kubernetes
  - --pool
  - inference-pool
  - --limit
  - "50"

- The

--limit

flag behaves the same in 2.x and 3.x — it controls how many flow runs a single worker will start simultaneously. Bottom line: - Defaults are intentionally minimal (no HPA, no per-worker cap). - When you do enable HPA, set both per-worker and (optionally) pool-level limits to avoid over-scaling and bursts. Links: - Helm charts repo: prefect-helm - Helm chart docs index: prefect-helm docs If you share your target concurrency and cluster sizing, I can suggest concrete numbers for

worker.config.limit

, HPA bounds, and a pool concurrency cap.

Short answer: the HPA in the prefect-worker chart scales strictly on Kubernetes resource metrics (CPU/memory). Hitting a worker’s

--limit 50

and having 50 flows running will not, by itself, trigger a scale-up unless the worker pod’s CPU/memory crosses the HPA targets. Why: the worker just submits/monitors runs; your flows execute in separate Kubernetes Job pods (or other infra). The worker’s resource usage usually stays low even when many flows are running elsewhere, so a CPU/memory-based HPA often won’t reflect backlog/throughput needs. Implications (Prefect 2.x and 3.x — same behavior): -

worker.config.limit

controls how many runs a single worker will start concurrently. It does not feed into HPA. - The Helm chart’s autoscaler uses CPU/memory targets only:

worker:
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 40
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80

- If you need scale-out when queue depth/backlog grows, use one of: - Pre-provision a higher replicaCount and cap throughput with

worker.config.limit

and a pool concurrency limit. - Use an autoscaler driven by external/custom metrics (e.g., KEDA) that scales on “pending/scheduled runs” for your work pool/queues via the Prefect API. - Shard queues and run multiple worker Deployments, each with its own HPA/limits. Recommended pattern: - Set a per-worker limit (e.g., 25–100). - Either: - Run a fixed number of replicas sized to your target throughput, and - Optionally set a work pool concurrency limit to cap total throughput - Or: - Introduce KEDA/custom metrics to scale by backlog instead of CPU/mem. Links: - Helm charts: prefect-helm - Helm docs: prefect-helm docs If you want, I can outline a KEDA ScaledObject that scales your worker Deployment based on the number of scheduled/pending runs in a specific work queue/pool.