Thanks for the details — assuming Prefect 3.x. Even with

PREFECT_WORKER_QUERY_SECONDS=5

, ~20s gaps can happen for several reasons. Here are the most common causes to check: - Polling jitter and backoff - Workers add ~30% jitter to the sleep interval, so a 5s interval can vary (~3.5–6.5s). - On transient errors (API/network), the loop backs off (doubles the interval up to a cap), which can easily produce ~20s sleeps until recovery. - References: critical_service_loop, clamped_poisson_interval, worker loop and worker settings. - Scheduler cadence on the server - The main scheduler loop runs roughly every 60s by default (with a faster loop for very recent deployments). New flow runs may not be scheduled immediately, causing visible gaps. - Reference: server scheduler settings. - Prefetch window - Workers only fetch runs scheduled up to

PREFECT_WORKER_PREFETCH_SECONDS

in the future (default ~10s). Depending on when a run is scheduled vs. the next poll, this can add several seconds. - Reference: worker get_and_submit_flow_runs. - Concurrency limits holding runs - Worker

limit

(local concurrency), work pool concurrency, deployment-level concurrency, or tag-based concurrency can keep runs in Scheduled until a slot frees up, often with a randomized wait. - References: worker limiter, concurrency v2, and tag concurrency handling in orchestration policies. - Work queue priorities - If a work pool has multiple queues with priorities, lower-priority queues may be deferred while higher-priority queues have pending work. - Reference: workers API. - Infrastructure spin-up time - Starting pods/containers/tasks (Kubernetes, Docker, ECS, etc.) can add seconds to tens of seconds between “submitted” and “running.” - This appears as a gap between runs even though polling is frequent. - API or DB slowness/transient errors - Slow responses or brief outages cause the polling loop to back off; check worker logs for errors/timeouts during polling. - Clock skew - If worker and server clocks differ, runs may appear “scheduled in the future,” delaying pickup until the next poll. A few quick checks to pinpoint the cause: - Inspect worker logs around the gap for signs of backoff or errors, and to see actual polling timestamps. - Verify

PREFECT_WORKER_PREFETCH_SECONDS

and your scheduler cadence. - Check worker

limit

, work pool/deployment/tag concurrency settings, and whether multiple queues with priorities are in play. - Look at infra start times (e.g., pod/container/task startup) around each run. - Confirm time sync on worker and server hosts. If you can share: - Prefect version and whether you’re on Cloud or self-hosted Server - Worker type/infra, work pool config, and any concurrency limits - A snippet of worker logs around the 20s gaps …I can help narrow it down quickly.

Thanks — that extra context helps. From your timestamps, there’s ~26s between the previous run’s state update (104848 set_state 201) and the next run’s process starting (104914 Sentry init). With a Process worker and no explicit limits, the most likely causes are: Most likely in your setup (self‑hosted Server on K8s, Process worker) - Server scheduler cadence - The server’s main scheduler loop runs every ~60s by default. That can introduce 0–60s (avg ~30s) of latency before a newly due run is actually created/scheduled for pickup. A 20–30s gap matches this pattern. - Tuning: set

PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS

lower (e.g., 10–15) on your server deployment to reduce scheduling latency. - Ref: server scheduler settings - Worker polling jitter/backoff - Even with

PREFECT_WORKER_QUERY_SECONDS=5

, the loop adds ~30% jitter, so actual sleeps vary (~3.5–6.5s). If there were transient API timeouts/slowness, the loop can temporarily back off to longer sleeps, contributing to gaps. - Refs: critical_service_loop, clamped_poisson_interval, worker loop, worker settings - Prefetch window - Workers only fetch runs scheduled within

PREFECT_WORKER_PREFETCH_SECONDS

(default ~10s). Depending on when a run becomes “due” vs. the next poll, this can add several seconds. - Ref: get_and_submit_flow_runs Other contributors to check - API/DB slowness or transient errors causing temporary backoff in the worker (look for warnings/errors around polling). - Clock skew between worker pod(s) and the server pod; skew can make runs appear “in the future.” - Process start overhead is usually small for Process worker, but your app init (e.g., Sentry, settings/Redis) adds a second or two after the worker has already submitted the run. How to pinpoint where the time is spent - Check the flow run timeline in the UI for these two runs: - When was the next run “Scheduled” on the server? - When did it become “Pending/Submitted”? - When did it become “Running”? - This will show if the delay is in scheduling vs. worker pickup vs. process startup. - Enable DEBUG logs on the worker to see actual poll timestamps and any backoff: - Look for lines like “Querying for flow runs” and any warnings/errors between polls. - Confirm effective settings:

prefect config view --show-defaults --show-sources

Verify

PREFECT_WORKER_QUERY_SECONDS=5

and consider:

export PREFECT_WORKER_PREFETCH_SECONDS=5

- If using self-hosted Server (your URL suggests K8s), reduce scheduler loop: - Set on the server pods:

PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS=15

Then restart the server deployment. If you can share: - Whether these runs are created by a deployment schedule or an automation/trigger - A screenshot (or timestamps) from the run timeline showing Scheduled → Submitted → Running - Any worker DEBUG logs around the gap …I can help identify the exact bottleneck and recommend a precise fix.