Thanks for the details — a few things jump out that likely explain the Pending/Late/Crash behavior. High-level diagnosis - You’re massively oversubscribing that 4‑core VM. With a Process work pool and 5 workers each started with

--limit 10

, you’re allowing up to 50 concurrent flow runs on a single 4‑CPU host. Each flow can also run tasks in parallel (default ThreadPool), so actual thread/process count can be far higher. That will cause 1) scheduling jitter/late starts, 2) Pending runs waiting for a slot, and 3) Crashes from OOM/CPU starvation. - Your pasted commands show the same worker name used repeatedly. Ensure each worker has a unique `--name`; reusing the same name can cause confusing handoffs/reconnections. - If many deployments fire on the same 5‑minute boundary, you’re creating a thundering herd that further amplifies the above. What I’d change first (on your current Process setup) 1) Right-size global and per-worker concurrency - Cap pool-wide concurrency

prefect work-pool set-concurrency-limit daily-agent-pool 10

Start lower (e.g., 6–10), then tune up as you observe stability. - Reduce worker limits and ensure unique names. On a 4‑core VM, I’d start with 2–3 total concurrent flows:

prefect worker start --pool daily-agent-pool --name daily-agent-worker-1 --limit 2 --prefetch-seconds 15
  prefect worker start --pool daily-agent-pool --name daily-agent-worker-2 --limit 1 --prefetch-seconds 15

This gives you 3 concurrent flow runs total. If stable, bump to 4–6 total. Avoid 5×10 on that host. 2) Limit intra-flow task concurrency - If you haven’t set a task runner, the default ThreadPool may create more threads than you expect. Pin it per flow:

from prefect import flow, task
  from prefect.task_runners import ThreadPoolTaskRunner

  @task
  def do_work(i):
      ...

  @flow(task_runner=ThreadPoolTaskRunner(max_workers=2))
  def my_flow():
      # keep parallelism modest on a small VM
      ...

For CPU-bound work you can consider ProcessPoolTaskRunner but still keep

max_workers

small (e.g., 2). 3) Stagger schedules - If 30+ deployments are scheduled “every 5 minutes” on the exact same second, offset them (e.g., minute 0,1,2,3,4) so not everything hits the queue at once. This reduces Pending/Late spikes and API/DB load. 4) Validate infra and logs - Check for OOM or limit messages in worker logs — if you see “flow run limit reached” often, that’s a sign the cap is working and you can scale up gradually if the box can handle it. - List backlogged/crashed runs:

prefect flow-run ls --state Pending --limit 50
  prefect flow-run ls --state Crashed --limit 50

- Inspect the pool config to confirm your limits took effect:

prefect work-pool inspect daily-agent-pool

5) Make sure the server/API tier is not your bottleneck - If your Prefect server (API + services) is sharing that same 4‑core VM with the workers, move it to a separate VM or scale it up. Spiky 5‑minute bursts can saturate the API and DB. Your Postgres is already separate — that’s good — but ensure it has headroom (connections, CPU, memory, IOPS). About ACI (Azure Container Instances) - Yes, in Prefect 3.x there is an ACI worker via the prefect-azure integration. It’s a good fit if you want isolation per run and elastic scale without managing a larger VM fleet. - Steps: 1) Install the integration:

pip install prefect-azure

2) Create an ACI work pool (after installing the integration, you’ll have the

azure-container-instance

pool type available):

prefect work-pool create azure-aci --type azure-container-instance

3) Configure the pool’s base job variables (image, cpu, memory). Example for deployments:

work_pool:
       name: azure-aci
       job_variables:
         image: <http://myregistry.azurecr.io/my-image:latest|myregistry.azurecr.io/my-image:latest>
         cpu: 2.0
         memory: 4.0

4) Provide credentials via a Prefect block (service principal) or use a managed identity on the VM that runs the worker. 5) Start a worker that points at the ACI pool (you can still cap concurrency here too):

prefect worker start --pool azure-aci --name aci-worker-1 --limit 5

- Then gradually move busy deployments to the ACI pool. Keep a sane pool-wide limit (e.g., 10–20) to control cost and API load. Why these changes help - Reducing

--limit

and setting a pool-wide concurrency limit prevents the VM from thrashing and reduces Pending/Late jitter. - Limiting task runner concurrency stops each flow from oversubscribing the host internally. - Staggered schedules remove the “thundering herd.” - Moving heavy or spiky workloads to ACI isolates resource usage and removes the small-VM bottleneck entirely. A few clarifying questions to tune next steps - Are your Prefect server/API services running on the same 4‑core VM as your workers? - Roughly how much memory does the VM have, and do you see OOM kills or swap activity during peaks? - Are your workers’ names unique? Your example repeats the same name; if that’s how you actually start them, we should fix that. - Do your flows do parallel work inside (e.g., map/submit many tasks at once)? If you share those, I can recommend concrete pool/worker limits and a rollout plan to ACI tailored to your load.

Got it — a few likely culprits can cause big disk reads and crashes, depending on which VM is spiking. Can you confirm which box is affected (the Prefect server VM, the worker VM that submits to ACI, or your database host)? Also, are you self‑hosting Prefect Server (SQLite vs Postgres) or using Prefect Cloud? Here are the most common causes and quick mitigations: If the VM is running the Prefect Server (self-hosted) - Likely cause: SQLite under load. Prefect logs are stored in the DB; high log volume and queries can hammer the

~/.prefect/prefect.db

file and cause heavy disk I/O. - Mitigations: - Move to Postgres: set

PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@host:5432/prefect>

and restart the server. This shifts disk I/O to your DB server and handles concurrency better. - Reduce DB churn from logs: - Temporarily disable log shipping to the API:

PREFECT_LOGGING_TO_API_ENABLED=false

- Lower verbosity:

PREFECT_LOGGING_LEVEL=INFO

(or

WARNING

) - Docs: - Self-hosted guide: Self-hosted server - Settings: Settings reference If the VM is the Prefect worker host for an ACI pool - Prefect itself is very light on disk here. The worker mainly polls and submits jobs to Azure; ACI pulls images and runs your code in Azure, not on the worker VM. - Things to check: - File logging: Ensure you haven’t added a file log handler that’s writing/rotating large logs on disk. Prefect defaults to console+API logging only. - Log streaming: Keep ACI

stream_output

off (default). It’s network-bound, but if you’ve scripted anything to capture logs to files locally, that can churn disk. - Anything else on that VM (Docker pulls, antivirus scans, other services) is more likely the culprit. If you’re still running flows on the VM (not yet ACI) - from_source downloads:

flow.from_source(...)

pulls code into a temp dir on every run and can cause heavy I/O with large repos or artifacts. - Mitigate by using deployment pull steps so code is pulled once during deployment, or bake code into your image. - Guide: Store flow code - Concepts: Deployments Result persistence - By default in Prefect 3, results are not persisted (

PREFECT_RESULTS_PERSIST_BY_DEFAULT=false

), so this shouldn’t create local disk I/O unless you explicitly enabled it or configured a file-based storage block. - If you need persistence, point to remote storage (e.g., Azure Blob) via

PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK

to avoid local disk. ACI notes (for when you switch) - Keep

stream_output=false

on the ACI worker (default). ACI log streaming is polling the Azure API (network), not local disk, but turning it on won’t help your VM’s disk profile. - ACI worker guide: ACI worker Fast triage you can try now - If you’re self-hosting the server on that VM: - Disable API logging temporarily:

PREFECT_LOGGING_TO_API_ENABLED=false

- Drop log verbosity:

PREFECT_LOGGING_LEVEL=WARNING

- If using SQLite, plan a quick move to Postgres via

PREFECT_API_DATABASE_CONNECTION_URL=...

- If it’s the worker VM: - Verify you’re not writing logs to files. - Confirm you’re not actually executing flows locally (e.g., using a process/docker pool) with large

from_source

pulls. If you can share: - Which process shows at the top during the spike (e.g.,

python

,

sqlite3

,

prefect-server

,

docker

), - Whether your Prefect server is SQLite or Postgres, - Whether the spiking VM is the server or the worker host, I can give you a more precise fix.