Short answer - Within a single pod, Prefect Server background services are async Python loops running in one process. They won’t meaningfully use multiple CPU cores in parallel. Giving the pod >1 CPU won’t translate to multi-core execution for these services. - Scaling them with multiple replicas is not recommended by default. If you do, you must enable a shared coordination backend (Redis via Docket) so only one instance of each perpetual service performs a given job. Without that, you risk duplicate scheduling/processing. Most installs keep backgroundServices at a single replica. How to scale safely (if you really need to) - Run background services as a separate deployment and point Docket at Redis so replicas coordinate:

backgroundServices:
  runAsSeparateDeployment: true
  replicaCount: 1        # keep 1 unless you’ve enabled Redis coordination
  messaging:
    docket:
      url: "<redis://my-redis-host:6379/0>"  # or set PREFECT_SERVER_DOCKET_URL

- If you then raise replicaCount above 1, ensure all API pods run with services disabled (the Helm chart does this automatically when runAsSeparateDeployment: true) and that all backgroundServices replicas share the same Redis Docket URL. This prevents duplicate schedulers/monitors. Links - Helm chart (prefect-server) docs and values: prefect-helm chart - Self-hosted/HA guidance (coordination, Redis, multi-server): Self-hosted Prefect docs If you can share whether you’re running backgroundServices inside the API pods or as a separate deployment, I can suggest the exact values and readiness checks to use.

Great — since you’re already running backgroundServices as a separate pod, here’s how to split them further. Can I run specific services in separate pods? - Not directly via a built-in Helm toggle. The prefect-helm chart runs “all enabled services” in a single background-services Deployment. - You can achieve separation by running multiple background-service pods, each with only the desired services enabled via environment variables. How to do it safely - Use Docket with Redis so multiple pods coordinate and you don’t get duplicate scheduling/processing: - Set

PREFECT_SERVER_DOCKET_URL=redis://<redis-host>:6379/<db>

(or use the Helm values at

backgroundServices.messaging.docket.url

). - Start each pod with

prefect server services start

and selectively enable/disable services via env vars. Service toggles (use these in the pod’s env) - Enable/disable specific services with `PREFECT_API_SERVICES_<SERVICE>_ENABLED`: - SCHEDULER, LATE_RUNS, FOREMAN, PAUSE_EXPIRATIONS, REPOSSESSOR, DB_VACUUM, CANCELLATION_CLEANUP, EVENT_PERSISTER, TASK_RUN_RECORDER, TRIGGERS, EVENT_LOGGER - Example: scheduler-only pod:

- name: PREFECT_SERVER_DOCKET_URL
    value: <redis://redis:6379/0>
  - name: PREFECT_API_SERVICES_SCHEDULER_ENABLED
    value: "true"
  - name: PREFECT_API_SERVICES_LATE_RUNS_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_FOREMAN_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_PAUSE_EXPIRATIONS_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_REPOSSESSOR_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_DB_VACUUM_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_CANCELLATION_CLEANUP_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_TASK_RUN_RECORDER_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_TRIGGERS_ENABLED
    value: "false"
  - name: PREFECT_API_SERVICES_EVENT_LOGGER_ENABLED
    value: "false"

Two practical ways to implement this in Kubernetes - Keep the Helm chart for API + one backgroundServices deployment, and add extra custom Deployments for additional service-only pods (recommended). - Create a small Deployment using the same Prefect image and command, with the env block above. For example:

apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: prefect-background-scheduler-only
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: prefect-background-scheduler-only
      template:
        metadata:
          labels:
            app: prefect-background-scheduler-only
        spec:
          containers:
            - name: services
              image: prefecthq/prefect:3.6.2
              command: ["bash", "-lc", "prefect server services start"]
              env:
                # include the env block from the example above
                - name: PREFECT_SERVER_DOCKET_URL
                  value: <redis://redis:6379/0>
                - name: PREFECT_API_SERVICES_SCHEDULER_ENABLED
                  value: "true"
                - name: PREFECT_API_SERVICES_LATE_RUNS_ENABLED
                  value: "false"
                # ...etc
              resources:
                requests:
                  cpu: "500m"
                  memory: "512Mi"
                limits:
                  cpu: "1"
                  memory: "1Gi"

- Or, disable the chart’s backgroundServices deployment entirely and manage multiple custom service Deployments yourself (one for scheduler, one for late runs, etc.). This avoids redundancy but shifts management to your manifests. Notes and cautions - If you run more than one pod for the same service, use Redis-backed Docket; otherwise, you’ll get duplicate scheduling/cleanup work. - Splitting services is about operational isolation, not performance. These loops are lightweight and async; giving a pod multiple CPUs won’t make them parallelize across cores.

- When backgroundServices are separate, ensure your API pods run with services disabled (the Helm chart handles this automatically when using runAsSeparateDeployment). Docs/links - Helm chart: prefect-helm chart - Self-hosted guidance (multi-server, coordination): Self-hosted Prefect docs If you share: - your Redis/Docket setup, and - which services you want in which pods, I can draft exact Helm values or K8s manifests tailored to your cluster.

Got it — let’s zero in on what’s burning CPU in your backgroundServices pod and narrow it down quickly without breaking orchestration. A practical, low-risk triage plan 1) Turn on targeted diagnostics (temporarily) - This will surface which service loops are most active.

backgroundServices:
  env:
    - name: PREFECT_SERVER_LOGGING_LEVEL
      value: DEBUG
    - name: PREFECT_INTERNAL_LOGGING_LEVEL
      value: DEBUG
    - name: PREFECT_DEBUG_MODE
      value: "1"

- Then watch the pod logs and CPU together for 5–10 minutes: - kubectl logs -f <background-pod> - kubectl top pod <background-pod> --containers What to look for in logs: - Scheduler: lines like “Scheduled N runs” and how often it loops - Event persister: “Flushed N events”, “max flush retries” - DB vacuum: “Deleted N records” bursts at its interval - Triggers: steady event processing or reconnect churn 2) Identify the culprit by toggling services (one at a time) - Roll a pod restart each time and watch CPU for a few minutes. - Start with services most likely to cause spikes (in order): - DB vacuum - Triggers - Event persister - Use these toggles:

# Disable DB vacuum temporarily (safe to test)
- name: PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED
  value: "false"

# If you are not using Automations/Triggers, test disabling:
- name: PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED
  value: "false"

# If you’re not relying on event streams, test disabling:
# (CAUTION: events won’t be persisted while disabled)
- name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED
  value: "false"

- If CPU drops when one is disabled, that’s your driver. 3) Tuning knobs once you’ve found the driver - Scheduler (reduce activity frequency/batch size)

- name: PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS
  value: "120"               # default ~60; slow it down
- name: PREFECT_SERVER_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE
  value: "50"                # default ~100
- name: PREFECT_SERVER_SERVICES_SCHEDULER_INSERT_BATCH_SIZE
  value: "250"               # default ~500

- DB vacuum (make it lighter/rarer)

- name: PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS
  value: "14400"             # default ~3600; run every 4h
- name: PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE
  value: "100"               # default ~200

- Triggers (if you need them but they’re too chatty)

# Mostly event read loop behavior; keep defaults low or disable if unused
- name: PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED
  value: "true"

- Event persister (balance CPU vs. flush frequency)

- name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE
  value: "50"                # default ~20
- name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL
  value: "10"                # default ~5

4) Quick deep-dive if logs aren’t conclusive - Profile the process for 30–60s with py-spy to see hot code paths:

kubectl exec -it <background-pod> -- sh -lc '
pip install --no-cache-dir py-spy && \
py-spy top --pid 1 --rate 25 --duration 30
'

- If PID 1 isn’t the server process, run

ps -o pid,cmd

first and attach to the right PID. - You can also record a flamegraph:

py-spy record -o /tmp/services.svg --pid <pid> --duration 60

Copy it out with

kubectl cp

and inspect. 5) DB checks (common hidden cause) Even at low run volume, heavy queries (e.g., scheduler) can spike CPU if the DB is under-provisioned or missing indexes. - Check Postgres CPU during spikes. If Postgres is pegged, consider: - Up CPU/IO for Postgres - Verify indices/migrations are up-to-date (Helm deploys current schema) - Look for long-running queries: pg_stat_activity / pg_stat_statements - If your events table is huge and you don’t need long retention, raise vacuum interval and/or shorten event retention policies. Why this approach works - Your load (~2.5 runs/min, 10–30s each) is modest; services shouldn’t be hot all the time. The usual culprits are: - DB vacuum bursts (periodic spikes)

- Triggers/event loops reprocessing or reconnecting - Scheduler set to very aggressive loop/batches - Database resource contention If you can share: - Whether you use Automations/Triggers - Whether you rely on events persistence for downstream tooling - Your Postgres and Redis (Docket) setup and resource limits I can suggest a minimal set of toggles and tuned defaults for your exact use case. Docs for reference - Helm chart: prefect-helm chart - Self-hosted/HA guidance: Self-hosted Prefect docs

Thanks for the details — that helps. You’re right: in 3.6.2 there is no DB vacuum service, so we can rule that out. Given your workload and setup (a few automations, events persistence OK but not required, no Docket), the usual CPU suspects in the backgroundServices pod on 3.6.2 are: - Triggers (automations) service: can churn if the PG LISTEN/NOTIFY loop is reconnecting or if it’s processing more events than expected - Event persister: steady small flushes or retry loops if inserts fail - Scheduler: tight loop/batch settings (less likely at your volume, but possible) - Late runs / foreman: lightweight, but misconfig or frequent errors can spin A focused plan to identify the culprit 1) Turn on targeted diagnostics (temporarily) - This will surface which loop is hot without flooding everything.

backgroundServices:
  env:
    - name: PREFECT_SERVER_LOGGING_LEVEL
      value: DEBUG
    - name: PREFECT_INTERNAL_LOGGING_LEVEL
      value: DEBUG
    - name: PREFECT_DEBUG_MODE
      value: "1"

- Tail logs and CPU together for ~5–10 minutes: - kubectl logs -f <background-pod> - kubectl top pod <background-pod> --containers What to look for: - Triggers: frequent “pg_notify reconnect” / heartbeat errors or constant event handling even when idle - Event persister: “Flushed N events” very frequently or “max flush retries”/DB error loops - Scheduler: “Scheduled N runs” every loop and how often it loops 2) Quick A/B toggles to isolate - Since events persistence isn’t mandatory, first test disabling the persister (automations still work because triggers consume live events; only persistence stops):

- name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED
  value: "false"

If CPU drops noticeably, the persister is your driver. If not: - Briefly disable triggers during a maintenance window to test impact (this pauses automations while disabled):

- name: PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED
  value: "false"

If CPU drops, it’s likely the trigger loop or PG LISTEN/NOTIFY churn. - If neither affects CPU, try slowing the scheduler just to test:

- name: PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS
  value: "120"     # default ~60
- name: PREFECT_SERVER_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE
  value: "50"      # default ~100

3) Tune the culprit - If Event Persister is hot, try fewer flushes with larger batches:

- name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE
  value: "50"      # default ~20
- name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL
  value: "10"      # default ~5

Also confirm the DB isn’t rejecting inserts (look for transient errors). - If Triggers is hot: - Confirm Postgres is reachable and stable; if LISTEN/NOTIFY is dropping, you’ll see reconnect loops (high CPU). - If your automations are few and simple, consider keeping triggers enabled but ensure the database is not throttled and connection limits are sufficient. If reconnect churn persists, share logs and we can dig into the exact error pattern. - If Scheduler is hot: - Keep

LOOP_SECONDS

higher (e.g., 120) and reduce

DEPLOYMENT_BATCH_SIZE

(50) for your volume. 4) Optional: capture a quick CPU profile - If logs aren’t conclusive, a 30–60s py-spy sample will reveal hot code paths:

kubectl exec -it <background-pod> -- sh -lc '
pip install --no-cache-dir py-spy && \
ps -o pid,cmd && \
py-spy top --pid 1 --rate 25 --duration 30
'

If PID 1 isn’t the Prefect process, attach to the correct PID from the ps output. 5) Sanity checks in Postgres - Check DB CPU during spikes; background services may be waiting on/hammering the DB. - If you can, enable pg_stat_statements and look for hottest queries. - Since you’ve set event retention to 1 day (on 3.6.2, no vacuum service exists), large event volume can still make persister busy; the A/B test above will confirm. Exact env names for 3.6.2 (most relevant) - Triggers:

PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED

- Event persister:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED

,

..._BATCH_SIZE

,

..._FLUSH_INTERVAL

- Scheduler:

PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS

,

..._DEPLOYMENT_BATCH_SIZE

,

..._INSERT_BATCH_SIZE

- Logging:

PREFECT_SERVER_LOGGING_LEVEL

,

PREFECT_INTERNAL_LOGGING_LEVEL

,

PREFECT_DEBUG_MODE

Links - Helm chart: prefect-helm chart - Self-hosted/HA guidance: Self-hosted Prefect docs If you can paste a short log excerpt during a CPU spike (especially lines from triggers and event persister) and confirm your Postgres resource limits, I can recommend a minimal tuning set tailored to your cluster.

Ha! I wish I could blast it through Slack, but I can’t play audio here. Queue it up and crank it: - YouTube search: The Who – Pinball Wizard - Spotify search: The Who – Pinball Wizard - Apple Music search: The Who – Pinball Wizard Thanks for the kind words — enjoy the riffs, and sleep well!

You got it — virtual speakers to 11 and The Who is blasting in my head right now. Pinballs are ricocheting everywhere. If you want to join me: - Spotify: The Who – Pinball Wizard - YouTube Music: The Who – Pinball Wizard Catch you next time!