Short answer: Prefect won’t “auto-queue” based on your cluster capacity. If a worker is free to submit runs, it will create Kubernetes Jobs immediately. Whether those pods actually fit on your tainted/tolerated nodes is determined by Kubernetes scheduling rules — and Kubernetes only enforces capacity based on resource requests, not limits. If your jobs specify memory limits but little/no memory requests, the scheduler can place too many pods, leading to a thundering herd at 1am. Common reasons this happens - No global throttling at the Prefect layer - If your work pool has no concurrency limit, workers will submit as many runs as they can. Work queues are just filters; they do not throttle. - Resource requests vs limits mismatch in your job template - Kubernetes schedules by requests, not limits. If your pods only set

resources.limits.memory

(or set very small

resources.requests.memory

), the scheduler will over-subscribe the nodes. - Taints/tolerations or node selection not applied where you think - If tolerations/nodeSelector/affinity aren’t present in the base job template Prefect uses, pods might target the wrong nodes or remain Pending without ever queuing at Prefect. - Multiple workers or pools submitting in parallel - If you have several workers attached to the same pool, they’ll all submit concurrently unless you set a pool-level concurrency limit. - Thundering herd from schedules - Many deployments with the same 1:00 AM schedule will all become eligible at once; without a concurrency limit they’ll all be submitted. What to check and how to fix 1) Add a work pool concurrency limit to throttle submissions - This keeps runs “queued” at Prefect until capacity frees up.

prefect work-pool inspect "<your-pool>"
prefect work-pool set-concurrency-limit "<your-pool>" 10

Tune the value to roughly what your tainted nodes can handle. Docs: Work pools and concurrency limits 2) Ensure resource requests are set (not just limits) in the base job template - Your Kubernetes scheduler will only respect capacity with proper requests. Update the pool’s base job template so requests reflect real memory/CPU needs. - Get the default template to start from:

prefect work-pool get-default-base-job-template --type kubernetes --file base-template.json

- In the `pod`/container section, ensure:

"resources": {
  "requests": {
    "memory": "8Gi",
    "cpu": "4"
  },
  "limits": {
    "memory": "8Gi",
    "cpu": "4"
  }
},
"tolerations": [
  {
    "key": "workload",
    "operator": "Equal",
    "value": "data",
    "effect": "NoSchedule"
  }
],
"nodeSelector": {
  "nodegroup": "memory-heavy"
}

- Then apply the template to the pool:

prefect work-pool update "<your-pool>" --base-job-template base-template.json

Docs: Prefect + Kubernetes how-to and prefect-kubernetes integration 3) Confirm the right taints/tolerations and node selectors are actually on your Pod spec - Look at a generated Job/Pod from last night and verify

tolerations

, `nodeSelector`/`affinity`, and

resources.requests

are present as expected. 4) Reduce the thundering herd - Stagger schedules or add offsets so all deployments don’t become due at 1:00 AM. - Optionally split workloads into multiple pools (with distinct taints/node selectors) and set a concurrency limit per pool. 5) If needed, use task-level concurrency as a guardrail - If many tasks inside each flow are heavy, you can tag them and cap concurrency:

prefect concurrency-limit create k8s-heavy-tasks 5

Note: This controls task runs, not flow runs. Next steps / info that would help - Prefect version (assuming 3.x; correct me if not) - Output of: -

prefect work-pool inspect "<your-pool>"

- One Kubernetes Job/Pod YAML from a run (to check

resources.requests

,

tolerations

, and

nodeSelector

) - How many workers were running and which pools they poll - Whether you already have a work pool concurrency limit With that, I can help you set a safe pool limit and correct the job template so Kubernetes correctly queues based on node memory.

Short answer: you don’t need the OS bases to match, and you don’t need exact version lockstep across all components. What matters is keeping everything on the same major Prefect version (3.x) and upgrading in the right order. Guidelines for Prefect 3.x: - Major version: Keep server, workers, and your flow runtime (the Prefect version inside your job pod) on the same major version (3.x). - Minor/patch versions: Exact matches aren’t required. In practice, older clients (workers/flow runtime) talking to a newer server generally work. The risky direction is a newer client talking to an older server (can lead to 422/validation errors if the client sends fields the server doesn’t know yet). - Upgrade order: Upgrade server first → workers next → flow runtime images last. This minimizes compatibility issues. - Base images/OS: The Debian base used by the Prefect server image has no impact on what your job images use. Your job pods can stay on an older Debian as long as they can run Python and your dependencies, including the

prefect

Python package. - Practical pinning: If possible, pin workers and your job images to the same minor as your server (e.g., server 3.11.x → workers and job images use prefect==3.11.*). If you can’t, being within a couple of minors typically works—just test. Common symptoms of a mismatch: - New client → old server: API 422 errors, “unexpected/unknown field” in request payloads. - Old client → new server: Usually fine; you just won’t use new features until you bump the client. Example: keeping your job image on an older Debian while aligning Prefect:

# your flow runtime image
FROM debian:bookworm-slim  # or whatever you need
RUN apt-get update && apt-get install -y python3-pip && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir "prefect==3.11.*"  # match your server’s minor if you can

Docs/refs: - Prefect Server concepts: Prefect 3 Server concepts - Prefect 3 GA announcement (versioning context): Blog: Prefect 3 GA If you share: - Prefect server version - Worker version(s) - Prefect version inside your job image(s) …I can suggest a concrete upgrade plan and confirm whether you’re likely to hit any compatibility issues.