Thanks for the detailed report — this is a common point of confusion in HA/self‑hosted setups, especially with Redis messaging. What’s happening (and why you see a delay) - The flow graph in Prefect 3.x is built from the database, not directly from events. By design, the graph endpoint excludes PENDING runs. A task/subflow won’t appear in the graph until it transitions to RUNNING (or a terminal state). For short flows with a single task/subflow, there can be a visible gap between the flow-run starting and the first non‑PENDING state — during that time, the graph looks empty. - Additional latency can come from the event pipeline and the UI’s real‑time updates in HA: - Event persister batches writes (defaults can add up to ~5s before events are flushed). - In multi‑API-server deployments, WebSocket updates require sticky sessions at the load balancer. Without stickiness, real-time updates can be dropped on reconnect and you’ll only “catch up” when the UI refetches. Quick checks to confirm - Compare timestamps: when does the first task/subflow enter RUNNING vs when it first appears in the graph? If they line up, you’re seeing the “PENDING is hidden” behavior plus any batching/websocket delay. - Check worker logs: how long between flow-run creation and task pickup? If workers are a bit backlogged, that time will appear as an empty graph. - Open the browser dev tools Network tab and watch the WebSocket to

/api/events/out

. It should stay connected and receive messages. If it frequently reconnects and you’re behind a load balancer without sticky sessions, you’ll see delayed/missing live updates. - On your API pods, enable DEBUG logs and look for event persister messages about flushes and queue capacity. Mitigations and tuning - Reduce event batching latency (on your API pods): -

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1

(default is 5) -

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=10

- Optionally increase throughput:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=50-100

and

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000

- Ensure full HA wiring (critical): - Use Redis for messaging (not in‑memory):

PREFECT_MESSAGING_BROKER=prefect_redis.messaging

and set

PREFECT_REDIS_MESSAGING_URL=redis://...

- Make sure Docket uses Redis (not memory):

PREFECT_SERVER_DOCKET_URL=redis://...

- Configure sticky sessions for your load balancer so WebSockets consistently hit the same API instance, and be sure to enable the Upgrade/Connection headers for WS. - If you still see gaps after tasks are clearly RUNNING, consider upgrading to the latest 3.x — there have been fixes that improve graph visibility and event timing since 3.6.23. If you can share a bit more, I can help pinpoint the exact bottleneck: - Approximate delay you observe (e.g., 3–10s vs 30–60s)? - Load balancer type and whether sticky sessions are enabled? - Values (if any) you’ve set for: -

PREFECT_MESSAGING_BROKER

,

PREFECT_REDIS_MESSAGING_URL

-

PREFECT_SERVER_DOCKET_URL

- Event persister envs (flush interval, read/batch sizes) - A snippet of worker logs around a run that shows when the first task transitions to RUNNING. In short: part of the delay is expected (nothing shows while everything is still PENDING), but you can usually trim the rest by tuning event persister settings and ensuring sticky WebSockets + Redis-backed services in HA.

Got it — thanks for clarifying that the delay happens even when the flow run is already completed. Most-likely causes for “post-completion” delays - Graph query filtering: The graph endpoint excludes nodes if all of these are null:

flow_run.start_time

,

flow_run.expected_start_time

,

task_run.start_time

,

task_run.expected_start_time

. Very short/cached tasks or subflows can complete without a RUNNING phase and end up with null start_time. Those nodes won’t show in the graph until timing fields are present, which can look like a delay. - UI refresh path in HA: The initial graph is fetched via REST, but the UI also relies on real-time updates. In HA with Redis and a load balancer, if WebSocket connections aren’t sticky or events are batched, the UI may not refresh immediately after completion — you’ll see the graph “catch up” a bit later when the next refetch happens. - Event batching: The event persister batches (default flush ~5s). While the graph is DB-driven, other UI panels and triggers for refetching can be delayed if events land a few seconds later. - Clock skew/timezone edges: If any client/UI code or intermediaries apply a “since” filter with a time near “now”, skew between browser, API servers, and DB can exclude just-finished tasks until a subsequent refetch. Make sure all servers are NTP-synced. Fast way to pinpoint where the delay comes from 1) Hit the graph endpoint directly right after the run completes (bypasses UI/WebSocket):

curl -sS -H "Authorization: Bearer <TOKEN>" \
  "<YOUR_API_URL>/api/flow_runs/<FLOW_RUN_ID>/graph-v2?since=0001-01-01T00:00:00Z" \
  | jq '.nodes | length, .edges | length'

- If this returns the expected nodes/edges immediately, the backend is fine and the delay is in UI/event refresh. - If nodes are missing here, it’s likely the timing-field filter (e.g., null start_time/expected_start_time). 2) Compare against the task list for the same flow:

curl -sS -H "Authorization: Bearer <TOKEN>" \
  "<YOUR_API_URL>/api/task_runs/filter" \
  -H "Content-Type: application/json" \
  -d '{"flow_runs":{"id":{"any_":["<FLOW_RUN_ID>"]}}}' \
  | jq 'length'

- If the task list count > graph node count right after completion, you’re likely hitting the graph timing-field filter. 3) Confirm HA/WebSocket behavior - Ensure sticky sessions at the load balancer for the API (WebSockets to

/api/events/out

must consistently hit the same API instance). - In the browser dev tools, confirm the WS stays connected and receives messages immediately after completion. 4) Tune event batching (reduces visible delays in the UI) - On your API pods: -

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=10

- Optionally:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=50-100

,

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000

Recommendations - Verify the direct graph-v2 call with a “since” far in the past. If nodes are missing there right after completion, you’re almost certainly running into the timing-field filter. This is most visible on very short or cached tasks/subflows that don’t record a RUNNING start_time. - Ensure HA wiring is fully Redis-backed and sticky sessions are enabled so the UI gets timely real-time updates. - Consider upgrading to the latest 3.x — there have been fixes that improve event timing and graph-related behavior since 3.6.23. - Make sure all API servers and your DB are time-synced (NTP); even small skew can cause “just-finished” items to be filtered by time-based queries in some views. If you can share: - How long the delay is (e.g., ~5–10s vs 30–60s), - Whether the direct graph-v2 call shows nodes immediately, - Your load balancer type and whether stickiness is enabled, - Event persister env vars you’re using, …I can help zero in on the exact bottleneck and suggest precise config changes.

@Marvin - delay is more than 15 mins

15+ minutes is far beyond anything the Prefect event batching or UI polling would cause. That points to an environmental/read-after-write or caching issue rather than Prefect itself. Top likely causes for a 15‑minute delay 1) Database read-after-write lag via replicas or a DB proxy - If your API servers read from a replica (or a proxy that load‑balances to readers) but write to the primary, the UI’s REST calls can show “old” data until the replica catches up. A 15‑minute delay is exactly what we see when a replica lags. - Check your `PREFECT_API_DATABASE_CONNECTION_URL`: - It must target the writer/primary (e.g., RDS writer endpoint), not a reader/replica or a load‑balanced read pool. - If you use PgBouncer or a DB proxy, ensure it does not route some connections to replicas, and prefer session pooling for read‑after‑write consistency. 2) HTTP caching at the load balancer or CDN for /api responses - If a reverse proxy/CDN is caching GET responses for /api (e.g., 900s max‑age), the UI will keep seeing a stale graph and task list until the cache TTL expires (~15m). - In your browser Network tab, inspect the response headers for

/api/flow_runs/<id>/graph-v2

and

/api/task_runs/filter

. If you see Cache-Control with max‑age or s‑maxage around 900, that’s your culprit. - Mitigation: Disable caching for the /api path at your proxy/CDN or set

Cache-Control: no-store

on those routes. 3) UI cache + missing real‑time invalidation (sticky sessions) - The UI uses a client cache and real‑time events to invalidate and refetch. If WebSockets aren’t sticky across API pods, the client may miss invalidations and only refetch on a longer interval (often ~15m in some views). - Test: After a flow completes, do a hard reload of the run details page. If the graph shows up immediately after a hard reload but not “live”, this is a WebSocket stickiness/invalidation issue. - Ensure sticky sessions at the load balancer and that WebSocket upgrade headers are enabled for

/api/events/out

. How to isolate the root cause quickly 1) Bypass the UI and hit the API immediately after a run completes - Graph:

curl -sS -H "Authorization: Bearer <TOKEN>" \
  "<YOUR_API_URL>/api/flow_runs/<FLOW_RUN_ID>/graph-v2?since=0001-01-01T00:00:00Z" \
  | jq '.nodes | length, .edges | length'

- Task runs list:

curl -sS -H "Authorization: Bearer <TOKEN>" \
  "<YOUR_API_URL>/api/task_runs/filter" \
  -H "Content-Type: application/json" \
  -d '{"flow_runs":{"id":{"any_":["<FLOW_RUN_ID>"]}}}' \
  | jq 'length'

- Outcomes: - If these show the expected data immediately, the backend is fine; look at WebSocket stickiness and/or HTTP caching at the proxy. - If these are missing data right away, check your DB topology for replicas and confirm the API is reading from the writer. 2) Verify database topology - Are you using RDS/Aurora with reader endpoints, or any read replicas? - Is

PREFECT_API_DATABASE_CONNECTION_URL

pointing at a writer/primary endpoint? - If using PgBouncer, try session pooling and ensure it routes reads to the writer (or a proxy that provides read‑after‑write consistency). 3) Check your reverse proxy/CDN - Ensure there’s no caching for

/api/*

. If you’re using NGINX/CloudFront/Cloudflare, disable caching or set

Cache-Control: no-store

for API responses. - Look for any microcaching configured on your NGINX location for

/api

. 4) Confirm HA/WebSocket setup - Enable sticky sessions on your LB for the API so the WS to

/api/events/out

remains on the same instance. - In browser dev tools, ensure the WS stays connected and receives events promptly when runs complete. Notes - Event persister batching (even at defaults) accounts for seconds, not 15+ minutes. - Time skew can exacerbate issues but usually won’t create a consistent 15‑minute gap. If you can share: - Your DB connection string target (writer vs reader), and whether you use PgBouncer/DB proxies

- The load balancer/proxy/CDN in front of your API and whether caching is enabled for /api - Whether a hard reload makes the graph appear immediately - Results of the two curl calls above right after a run completes …I can help pinpoint the exact culprit and propose precise config changes.

Short answer: No — that PR is not related to your 15+ minute delay or to whether flow/task data is persisted to Postgres. What that PR does - PR #21264 adds a time-based checkpoint to the events WebSocket client to limit how much of the events stream is replayed on reconnect. It affects event streaming/backfill behavior only. - It does not change how flow runs or task runs are written to the database, nor how the flow graph is built. How Prefect persists what you see in the graph - Flow and task runs are written synchronously to Postgres by the API when they are created and when their states change. This is not event-driven. - The flow graph endpoint reads directly from Postgres. Events (and that PR) do not gate task/graph visibility. Given your 15+ minute delay and “data is not persisted into Postgres DB” That points to environment topology, not the events client: - Database read-after-write lag via replicas/proxies is the most common cause. - If your API pods write to the primary but read from a lagging replica, the UI (and your API calls) will “miss” data until the replica catches up — 10–20 minutes is a classic lag symptom. - HTTP caching at a proxy/CDN for /api responses (e.g., 900s TTL) can also produce a ~15 min delay. Quick isolation steps 1) Bypass the UI and hit the API right after a run completes:

# Graph (should return nodes immediately upon completion)
curl -sS -H "Authorization: Bearer <TOKEN>" \
  "<YOUR_API_URL>/api/flow_runs/<FLOW_RUN_ID>/graph-v2?since=0001-01-01T00:00:00Z" \
  | jq '.nodes | length, .edges | length'

# Task runs list
curl -sS -H "Authorization: Bearer <TOKEN>" \
  "<YOUR_API_URL>/api/task_runs/filter" \
  -H "Content-Type: application/json" \
  -d '{"flow_runs":{"id":{"any_":["<FLOW_RUN_ID>"]}}}' \
  | jq 'length'

- If these show expected data immediately, the backend is fine; look for HTTP caching or WS stickiness issues. - If these are missing right away and only show up 15+ minutes later, it’s almost certainly DB read-from-replica lag. 2) Check the DB directly on the writer/primary immediately after completion:

-- Replace with your IDs
SELECT id, name, state_type, start_time, end_time
FROM task_runs
WHERE flow_run_id = '<FLOW_RUN_ID>'
ORDER BY created DESC
LIMIT 20;

SELECT trs.task_run_id, trs.state_type, trs.timestamp
FROM task_run_states trs
JOIN task_runs tr ON tr.id = trs.task_run_id
WHERE tr.flow_run_id = '<FLOW_RUN_ID>'
ORDER BY trs.timestamp DESC
LIMIT 20;

- If these rows exist on the primary right away but not via your API/UI, your API is reading from a replica or a proxy that routes some reads to replicas. 3) Verify your DB connection target - Ensure

PREFECT_API_DATABASE_CONNECTION_URL

points to the writer/primary endpoint (e.g., AWS RDS writer), not a reader endpoint or a read-balanced proxy. - If using PgBouncer/DB proxies, prefer session pooling and ensure no read splitting that breaks read-after-write consistency. 4) Rule out HTTP caching on /api - Check response headers for

/api/flow_runs/<id>/graph-v2

and

/api/task_runs/filter

. There should be no Cache-Control max-age/s-maxage on those routes. Disable caching for

/api/*

in your LB/CDN. 5) WebSockets/sticky sessions - Not the cause of data not being persisted, but lack of stickiness can delay UI invalidation. Enable sticky sessions and WS upgrade headers for

/api/events/out

. If you can confirm: - Your exact

PREFECT_API_DATABASE_CONNECTION_URL

target (writer vs reader/proxy), - Whether the SQL above shows the rows on the primary right away, - Any proxy/CDN in front of your API and its caching rules, …I can help pinpoint the misconfiguration and suggest precise fixes.

@Marvin - When checked the data for the flow, task runs details are not persisted to DB. In Server logs, events are published, but background service didn't received the event.

Sorry, I encountered an error while processing your request. Please try again.

Thanks — that detail helps a lot. Two key points to separate: - Task/flow runs in Postgres: These are written synchronously by the API when the engine creates runs and sets states. Events are not required for that. If task runs are missing from Postgres, the engine is not writing to the same API/DB you’re checking or the API is writing to a different DB. - Events pipeline: “Events are published but background service didn’t receive” indicates a messaging/broker misconfiguration in HA. That explains missing rows in the events table, but not missing task_runs. It does, however, hint that your HA services aren’t all pointed at the same shared backends. Most likely root causes in HA - API pods use different DB URLs. Some pods write to one Postgres, others read from another (or from a lagging replica). - Workers are pointed at a different Prefect API base URL than the one your UI/LB targets. - Messaging broker mismatch: one process publishes to in-memory or a different Redis; the background services listen on another broker or Redis DB. - Event services disabled on the API pods (persister/distributor not running). Targeted checks to run now 1) Verify DB writes vs reads - On each API pod, print the active DB URL:

prefect config view | grep PREFECT_API_DATABASE_CONNECTION_URL

- They must be identical across all API pods and point to the same writer/primary endpoint. - If you have read replicas/proxies, make sure the URL is the writer (not a reader endpoint). 2) Verify worker -> API path - On each worker pod:

prefect config view | grep PREFECT_API_URL

- Confirm workers point to the same API base URL your UI is using. If they point somewhere else (e.g., a different LB, a pod IP, or Cloud), they’ll write to a different API/DB than the one you’re inspecting. 3) Confirm messaging is truly shared (not in-memory) - On ALL API pods (and anywhere background services run), verify:

prefect config view | grep -E "PREFECT_MESSAGING_BROKER|PREFECT_REDIS_MESSAGING_URL|PREFECT_SERVER_DOCKET_URL|PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED|PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED"

- Recommended values in HA:

PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_REDIS_MESSAGING_URL=redis://<redis-host>:6379/0
PREFECT_SERVER_DOCKET_URL=redis://<redis-host>:6379/1
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=true

- If any pod shows an in-memory/default broker or a different Redis DB/index/URL, events will be “published” but never seen by the persister/distributor. 4) Check logs on the API pods for background services - You should see logs from the event persister and distributor starting up and connecting to Redis. If they’re missing or erroring, events will not persist. - Also check that the API logs show incoming requests for: -

POST /api/task_runs/

(task run create) -

POST /api/task_runs/<id>/set_state

(state transitions) - If these do not appear during your run, the workers are not targeting this API. 5) Direct DB verification (on the writer/primary) - Immediately after a run completes, run:

SELECT id, name, state_type, start_time, end_time
FROM task_runs
WHERE flow_run_id = '<FLOW_RUN_ID>'
ORDER BY created DESC
LIMIT 20;

SELECT trs.task_run_id, trs.state_type, trs.timestamp
FROM task_run_states trs
JOIN task_runs tr ON tr.id = trs.task_run_id
WHERE tr.flow_run_id = '<FLOW_RUN_ID>'
ORDER BY trs.timestamp DESC
LIMIT 20;

- If these rows are present on the primary, but the UI/API doesn’t show them, your API is reading from a different DB or a stale replica. - If these rows are not present on the primary, the worker/API path is not writing to this database. Why PR #21264 is not relevant

- Add time-based checkpoint to events websocket client only affects event stream replay windows on reconnect. It does not control database persistence of flow/task runs, and it will not cause 15+ minute visibility gaps or missing task_run rows. If you can share: - The values/output of the config checks above from one API pod and one worker pod, - Whether you’re using a DB proxy/replicas (and which endpoints your API uses), - A short snippet from the API logs during a run (showing whether

POST /task_runs

and

.../set_state

are received), …I can help pinpoint exactly where the path breaks and give you the exact changes to make.

Sorry, I encountered an error while processing your request. Please try again.

Short answer: You can run multiple API/background-service instances, but only if they all coordinate through the same shared backends. Running them across two Kubernetes clusters without a single shared Docket (leader election), shared messaging broker, and the same writer database will create split‑brain and can lead to lock contention and “deadlock”-like behavior. What “multiple instances” means in Prefect Server 3.x - Singleton services (must be coordinated via Docket): - Scheduler - Automations - Late-run detector - DB vacuum/cleanup These are designed to run on many API pods, but with only one active leader at a time. Leader election is performed via Docket. If each cluster uses a different Docket backend (or memory docket), both clusters will think they are the leader and compete on the same rows — that’s when you can see database deadlocks or thrashing. - Scalable services (multi-instance is okay if the broker is shared): - Event persister (reads events from the broker and writes to DB) - Event distributor (feeds WebSockets) These can be run on multiple API pods, but every instance must read from the same messaging broker/Redis. If each cluster points to a different broker (or one uses in‑memory), events will be “published” but never seen by the persister in the other cluster. Recommended patterns Option A (simplest, most reliable) - Run all Prefect API + background services in one cluster (HA with multiple replicas). - In the other cluster(s), run only workers/executors. - Avoids split‑brain and cross‑cluster lock contention entirely. Option B (multi-cluster API) - Only do this if all instances share these exact backends: - Postgres: the same writer/primary endpoint in PREFECT_API_DATABASE_CONNECTION_URL - Messaging broker: Redis URL identical everywhere (PREFECT_MESSAGING_BROKER=prefect_redis.messaging and PREFECT_REDIS_MESSAGING_URL=redis://…) - Docket (leader election): the same Redis URL (PREFECT_SERVER_DOCKET_URL=redis://…) and name (PREFECT_SERVER_DOCKET_NAME) across all API instances - Ensure low latency and stable connectivity to reduce lease churn. - Time sync via NTP across all nodes to avoid TTL/lease anomalies. If you don’t want to share Docket across clusters, disable singleton services on one entire cluster: - Set these on the “secondary” cluster API pods: - PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false - PREFECT_SERVER_SERVICES_AUTOMATIONS_ENABLED=false - PREFECT_SERVER_SERVICES_LATE_RUNS_DETECTOR_ENABLED=false - PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=none (or false if you use a boolean flag) - Generally keep event services enabled only where your broker is reachable; if you’re not sharing the broker, disable: - PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=false - PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=false Why you’re likely seeing deadlocks - Two schedulers (one per cluster) running without a shared Docket both try to transition the same runs. Some update paths don’t use SKIP LOCKED and can end up in circular waits — you’ll see “deadlock detected” in Postgres logs. - Two automation engines triggering the same artifacts/concurrency updates from each side can also create row lock contention. What to verify on your pods (both clusters) Run these on at least one API pod per cluster: - DB target: -

prefect config view | grep PREFECT_API_DATABASE_CONNECTION_URL

- Should be the same writer/primary endpoint for every API pod in both clusters. - Messaging: -

prefect config view | grep -E "PREFECT_MESSAGING_BROKER|PREFECT_REDIS_MESSAGING_URL"

- Must be identical across clusters if you expect shared event handling. - Docket (leader election): -

prefect config view | grep -E "PREFECT_SERVER_DOCKET_URL|PREFECT_SERVER_DOCKET_NAME"

- Must be identical across clusters if you want a single leader across all API instances. - Services toggles: -

prefect config view | grep PREFECT_SERVER_SERVICES_

- Confirm you haven’t unintentionally enabled singleton services in both clusters when Docket is not shared. Operational checks - Logs: look for “acquired lease” messages (e.g., for scheduler/automations). If both clusters report acquiring the same lease, your Docket backends differ or are partitioned. - Postgres logs: search for “deadlock detected” around the times these services run. This confirms contention from duplicate leaders. - If you must run across clusters and still see flapping, consider increasing the Docket lease TTL/heartbeat interval (if exposed in your version), and ensure network reliability/low latency to Redis. Bottom line - Yes, you can run multiple background service instances — but only if they all coordinate through the same Redis-backed Docket and messaging broker and the same Postgres writer. Otherwise, disable singleton services on all but one cluster to avoid deadlocks. - Given your symptoms (“background service didn’t receive event” and “deadlocks”), I strongly suspect mismatched Redis broker and/or Docket URLs between clusters, or different DB endpoints (writer vs reader). Verifying those config values will likely surface the root cause. If you share the config outputs above from one API pod in each cluster, I can suggest the exact changes to make.

Thanks for confirming Postgres and Redis are the same across clusters. Given your symptoms (deadlocks and “events published but background service didn’t receive”), this usually comes down to cross‑cluster coordination details rather than the raw endpoints. What to verify first (small mismatches cause big split‑brain) - Docket (leader election) must be truly identical everywhere: - Same URL including DB index and scheme (redis:// vs rediss://), host, port, credentials. - Same name: -

PREFECT_SERVER_DOCKET_URL

-

PREFECT_SERVER_DOCKET_NAME

(must match across clusters to elect a single leader) - Messaging broker must be identical: -

PREFECT_MESSAGING_BROKER=prefect_redis.messaging

-

PREFECT_REDIS_MESSAGING_URL

exactly the same (host, port, db index, scheme, credentials, query params) - Database URL must be identical and point to the writer: -

PREFECT_API_DATABASE_CONNECTION_URL

on all API pods must target the same writer/primary endpoint (not a reader/replica). - Time sync and network stability: - All nodes in both clusters should be NTP‑synced. - Latency/jitter to Redis can cause lease flapping; make sure both clusters have low‑latency, reliable paths to Redis. Why you can still see deadlocks even with “same Postgres and Redis” - If Docket name/URL differ even subtly (different Redis DB index or TLS scheme), each cluster will elect its own leader. Two schedulers/automations will compete on the same rows, which can produce Postgres “deadlock detected”. - If Redis connectivity is flaky from one cluster, that side may frequently “lose and re‑acquire” the lease, causing leadership thrash and more contention. “Events published but background service didn’t receive” - If any API instance is using a different messaging backend (e.g., default in‑memory) or a different Redis DB index, you’ll see “published” at the publisher but nothing consumed by the persister/distributor in the other cluster. - Also confirm the event services are actually enabled on the API pods that should be consuming: -

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true

-

PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=true

- Startup logs on those pods should clearly show connection to the Redis broker and periodic “flushed N events” messages. If you don’t see those, the persister isn’t running/connected. Recommended deployment pattern to stop the deadlocks now - Run singleton services in only one cluster. On the “secondary” cluster, disable them:

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false
PREFECT_SERVER_SERVICES_AUTOMATIONS_ENABLED=false
PREFECT_SERVER_SERVICES_LATE_RUNS_DETECTOR_ENABLED=false
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=none

- Keep event services enabled only if they point to the same Redis broker and are stable. If you suspect broker differences or network instability, temporarily disable on one cluster to avoid confusion:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true   # enable on primary
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=false  # disable on secondary (temporarily)
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=true          # primary
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=false         # secondary (temporarily)

Run these quick checks on one API pod in each cluster

prefect config view | grep -E "PREFECT_API_DATABASE_CONNECTION_URL|PREFECT_MESSAGING_BROKER|PREFECT_REDIS_MESSAGING_URL|PREFECT_SERVER_DOCKET_URL|PREFECT_SERVER_DOCKET_NAME|PREFECT_SERVER_SERVICES_"

- Paste the outputs (redact secrets) and I’ll spot any inconsistencies. Operational tips - Watch logs for “acquired lease” messages for scheduler/automations. You should see them only from one cluster if Docket is truly shared. - Check Redis keyspace/DB index: a common gotcha is one cluster using

/0

and the other

/1

. - Ensure your load balancer keeps WebSockets sticky for

/api/events/out

, but note this is about live UI updates — it won’t fix missing DB rows.

If you prefer to stay multi‑cluster with all services active - It can work, but you must: - Use the same Redis (URL, DB index, name) for Docket and broker on all instances. - Keep network latency low and clocks in sync to avoid lease flapping. - Consider increasing Docket lease TTL/heartbeat tolerance if you still see churn (if exposed in your version). - Even then, many teams choose to centralize the API/background services in one cluster and run workers in multiple clusters for simplicity and fewer edge cases. Share the config outputs and a short log snippet from each cluster at API startup (showing Docket/broker connection and any “acquired lease” lines), and I’ll give you concrete, line‑by‑line fixes.

Thanks — that makes sense. If you’re running only a single instance of the event persister under heavy load, it will build a backlog and you’ll see long delays before events land in Postgres. Key points and fixes - Scale out the event persister - The event persister can run on multiple API pods safely as long as all instances point to the same Redis broker. This is the primary lever to reduce backlog latency. - Increase your API Deployment replicas and keep

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true

on all of them. - Keep singleton services (scheduler, automations, late-run detector) coordinated with a single shared Docket or disable them on one cluster to avoid deadlocks (details below). - Tune the persister for throughput - These settings have the biggest impact on catch-up under load: -

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=100

(or higher) -

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=200-500

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1-2

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000-200000

- Validate Postgres can absorb the write volume (connection pool/proxy, IOPS, WAL throughput). If using PgBouncer, use session pooling for write-heavy workloads and raise pool sizes accordingly. - Keep Redis and Docket shared across all API instances - Messaging broker (events): -

PREFECT_MESSAGING_BROKER=prefect_redis.messaging

-

PREFECT_REDIS_MESSAGING_URL=redis://<redis-host>:6379/<db>

- Docket (leader election for singleton services): -

PREFECT_SERVER_DOCKET_URL=redis://<redis-host>:6379/<db>

-

PREFECT_SERVER_DOCKET_NAME=<same across all instances>

- If you run across two K8s clusters and don’t want to share Docket, disable singleton services on one cluster: -

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false

-

PREFECT_SERVER_SERVICES_AUTOMATIONS_ENABLED=false

-

PREFECT_SERVER_SERVICES_LATE_RUNS_DETECTOR_ENABLED=false

-

PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=none

- Observe and verify - Enable DEBUG logs on the API pods and watch for: - Event persister flush cadence and batch sizes - Warnings like “Event queue at X% capacity” - The backlog should begin to drain immediately after scaling/tuning; the “delay until persisted” should drop from minutes to seconds. - Note on flow/task run persistence - Flow/task runs and their states are written synchronously by the API (not by the event persister). If you still see missing task_runs in Postgres, double-check worker → API routing and that all API pods point to the same writer database. The event persister backlog only impacts the events tables and event-driven features (e.g., automations, events UI), not the core run/state rows. If you share: - Your current event rate (approx flows/min, tasks/min), - Current values for the four persister settings above, - API replica count and DB/proxy setup, …I can suggest concrete target values and a scaling plan to eliminate the 15+ minute lag.

Yes. In Prefect 3.6.24 (with Docket), you can run multiple background service instances safely — with a few important rules. What you can scale horizontally - Event Persister (events → Postgres): Safe to run on many API pods. This is the primary lever to eliminate your backlog/delay. - Events Distributor (WebSocket stream): Can run on many API pods; each serves its connected clients. Make sure your load balancer supports WebSocket upgrades and uses sticky sessions. Singleton services (one active leader at a time) - Examples: Scheduler, Automations, Late-run detector, and other cleanup/maintenance jobs. - These can run on every API pod, but there must be exactly one active leader at any time. Docket handles this (leader election) — as long as all instances share the same Docket backend and name. Hard requirements for multi-instance background services - Same Postgres writer endpoint on all API pods: -

PREFECT_API_DATABASE_CONNECTION_URL

must point to the writer/primary (not a read replica or mixed read/write proxy). - Same Redis broker for events on all API pods: -

PREFECT_MESSAGING_BROKER=prefect_redis.messaging

-

PREFECT_REDIS_MESSAGING_URL=redis://<host>:6379/<db>

- Same Docket (leader election) on all API pods: -

PREFECT_SERVER_DOCKET_URL=redis://<host>:6379/<db>

-

PREFECT_SERVER_DOCKET_NAME=<identical across all pods/clusters>

- If you span multiple Kubernetes clusters, network latency/jitter to Redis must be low and clocks NTP-synced; otherwise you can see lease flapping. If you keep two K8s clusters - Preferred: Run all API + background services in one “primary” cluster (multiple replicas), and run only workers in the “secondary” cluster(s). - If you must run API/background services in both clusters: - Ensure Docket URL/name and the Redis broker URL are exactly identical (host, port, DB index, TLS scheme, credentials). - If you cannot guarantee shared/low-latency Docket, disable singleton services in one cluster:

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false
    PREFECT_SERVER_SERVICES_AUTOMATIONS_ENABLED=false
    PREFECT_SERVER_SERVICES_LATE_RUNS_DETECTOR_ENABLED=false
    # Optional if you’ve enabled any cleanup services
    PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=none

- You can still scale the Event Persister across pods in the “primary” cluster to drain event backlogs quickly. Throughput tuning for the Event Persister - Scale API replicas up (each runs a persister instance). - Increase batching and read sizes to drain the queue faster:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
  PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=100
  PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=200-500
  PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1-2
  PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000-200000

- Ensure Postgres (and any pooler like PgBouncer) can handle the write concurrency. Operational checks - On at least one API pod per cluster:

prefect config view | grep -E "PREFECT_API_DATABASE_CONNECTION_URL|PREFECT_MESSAGING_BROKER|PREFECT_REDIS_MESSAGING_URL|PREFECT_SERVER_DOCKET_URL|PREFECT_SERVER_DOCKET_NAME|PREFECT_SERVER_SERVICES_"

- All URLs/names should match exactly (including Redis DB index and rediss:// vs redis://). - Logs: - You should see a single “acquired lease” for singleton services across all pods/clusters. - Event persister logs should show frequent flushes; if you see “Event queue at X% capacity,” increase concurrency/queue limits or add replicas. - For UI real-time updates, enable sticky sessions and WebSocket upgrade headers on your load balancer for

/api/events/out

. Bottom line - Yes — with Docket in 3.6.24, you can (and should) run multiple background service instances for HA and to handle heavy event load.

- Make sure all instances share the same Postgres writer, the same Redis broker, and the same Docket (URL + name). If running across two clusters and you cannot guarantee this, disable singleton services on one cluster to avoid deadlocks. - To remove your 15+ minute delay, scale out the Event Persister and tune its batch/read/flush settings as above.

Short answer: No — those flags do not exist. Verified CLI -

prefect server services start --safe-only

→ invalid -

prefect server services start --singleton-only

→ invalid The only options on that command are: -

--background

(or

-b

) -

--help

How to control which services start Use environment variables to enable/disable individual services before running `prefect server services start`: Common toggles - Scheduler:

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false

- Late runs detector:

PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=false

- Automations/Triggers:

PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=false

- Event persister:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true

← scale this up for your backlog - Task run recorder:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

- Cancellation cleanup:

PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false

- DB vacuum/cleanup: -

PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=events

(default) - or

flow_runs

, or

events,flow_runs

, or

none

Real-time streams - Events WebSocket stream:

PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=true

- Logs WebSocket stream:

PREFECT_SERVER_LOGS_STREAM_OUT_ENABLED=false

(usually off) Tips for your use case (heavy events backlog) - Scale out by running more API replicas; each runs an event persister instance. - Tune throughput: -

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=100

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=200-500

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1-2

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000-200000

- Keep singleton services safe across clusters with Docket; or disable them on one cluster entirely as needed. If you want, share the list from:

prefect server services ls

and the relevant env values you’re using, and I’ll suggest a minimal set to run per cluster.

You’re exactly on the right trail. In Prefect 3.6.24, the Task Run Recorder service defaults are very conservative and can cause large backlogs if left at batch size/read size of 1. What Task Run Recorder does - Consumes task-run events from the “events” broker topic. - Upserts rows into task_runs and inserts rows into task_run_states in batches. - It’s separate from the Event Persister; both can be scaled horizontally. Defaults that cause delays - read_batch_size = 1 - batch_size = 1 - flush_interval = 5s Under heavy load, this leads to one-message reads and one-row transactions with a 5s time-based flush — minutes of lag are common. How to tune it - Environment variables: -

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

-

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=50-200

-

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=50-200

-

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1-2

- Start with 50/50/1 and scale up based on DB capacity. If Postgres (or PgBouncer) can handle bigger transactions, move toward 100–200 for both batch sizes. - Monitor API logs for insert throughput and any warnings/errors; adjust if you see lock contention or slow commits. Scale horizontally - It’s safe to run multiple Task Run Recorder instances on multiple API pods. They join a consumer group and split work; duplicate processing is guarded by upsert/unique constraints. - Simply scale your API Deployment replicas; keep the same settings on each pod. Also tune Event Persister if you’re seeing event backlogs -

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=50-200

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=100-500

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1-2

-

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000-200000

Sanity checks - All API pods must share the same Redis broker URL and the same Postgres writer. - Verify with: -

prefect config view | grep PREFECT_REDIS_MESSAGING_URL

-

prefect config view | grep PREFECT_API_DATABASE_CONNECTION_URL

- Confirm the service is enabled: -

prefect server services ls

(look for task_run_recorder: enabled/running) If you share your current values and approximate event volume (task-run events/sec), I can suggest precise batch sizes to bring the delay down to ~1–2s.

Great question. In Prefect 3.6.24, you can run all of these services on multiple pods (even across multiple clusters), but there are two classes: 1) Horizontally scalable (multiple instances run concurrently and increase throughput) - TaskRunRecorder - ReactiveTriggers - Actions These are consumer-group–based services. Running more replicas across clusters will split the event/queue load as long as all instances share the same Redis broker and Postgres. 2) Leader-elected/singleton services (safe to run on many pods, but only one should be active at a time) - CancellationCleanup - Foreman - MarkLateRuns - FailExpiredPauses - Repossessor (caveat below) - RecentDeploymentsScheduler - Scheduler - Telemetry - ProactiveTriggers Notes and caveats - For the singleton group, it’s safe to run many replicas across clusters only if all instances share the exact same Docket backend (Redis) and name. Docket will elect a single active leader. If Docket is not shared, you’ll get split-brain and potential deadlocks. - Repossessor requires shared lease storage across all instances (Redis-backed concurrency lease storage). If one cluster uses in-memory lease storage, do not run multiple instances; make sure lease storage is external and shared. - For the horizontally scalable services (TaskRunRecorder, ReactiveTriggers, Actions), ensure: - Same Redis broker URL everywhere (PREFECT_MESSAGING_BROKER=prefect_redis.messaging and identical PREFECT_REDIS_MESSAGING_URL) - Same Postgres writer endpoint everywhere - Consider tuning batch sizes/flush intervals for throughput If you cannot share Docket across clusters, disable the singleton services on the secondary cluster:

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=false
PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=false
PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=none
# And any others you’ve enabled (e.g., Foreman, ProactiveTriggers, Repossessor)

Summary - Multi-instance, throughput-scaling: TaskRunRecorder, ReactiveTriggers, Actions - Multi-instance, leader-elected (one active at a time via Docket): all the rest (including Scheduler) - Across multiple clusters: fully supported if you share the same Redis broker and the same Redis-backed Docket (URL + name) and point all instances at the same Postgres writer.

Perfect — if all API/background-service pods across both clusters share the same Postgres writer, the same Redis broker, and the same Redis-backed Docket URL/name, then you can safely run multiple instances of the background services. Docket will elect a single active leader for the singleton services, and the consumer-group services will scale linearly with more replicas. Given your delay is from Task Run Recorder processing with tiny batches, here’s a concrete plan: 1) Scale out the Task Run Recorder (consumer-group, safe to run on many pods) - Add more API replicas (each runs a recorder instance). - Tune batching to drain the queue quickly:

# Task Run Recorder
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=50-200
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=50-200
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1-2

# Event Persister (often the other bottleneck)
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=50-200
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=100-500
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1-2
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000-200000

- Start at 50/50/1 for recorder, 100/200/1 for persister; raise gradually if Postgres and Redis keep up. 2) Keep singleton services enabled everywhere (Docket will ensure one leader) - Scheduler, RecentDeploymentsScheduler, MarkLateRuns, CancellationCleanup, FailExpiredPauses, Foreman, ProactiveTriggers, Telemetry, Repossessor. - Since you share the same Redis-backed Docket URL and name across clusters, only one instance will be active at any time. If you ever see leadership flapping (frequent acquire/release), it usually points to network latency/jitter to Redis. 3) Optional: centralize event-heavy services - If cross-cluster network latency to Redis/DB is non-trivial, consider running higher replica counts of Task Run Recorder and Event Persister in the cluster closest (network-wise) to Redis/DB. Keep at least one replica in the other cluster for HA. 4) Verify at runtime - On any API pod:

prefect server services ls
prefect config view | grep -E "PREFECT_API_DATABASE_CONNECTION_URL|PREFECT_REDIS_MESSAGING_URL|PREFECT_SERVER_DOCKET_URL|PREFECT_SERVER_DOCKET_NAME|PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER|PREFECT_SERVER_SERVICES_EVENT_PERSISTER"

- In API logs, you should see frequent recorder/persister flushes and no “queue at X% capacity” warnings once tuned. - Measure end-to-end: time from a task starting to when its state appears in Postgres should drop to ~1–2s with the above settings under load. 5) Postgres readiness - Ensure your DB/pooler can handle the increased batch sizes and concurrent transactions (e.g., PgBouncer session pooling, higher max connections, adequate IOPS/WAL). If you see lock contention or slow commits, reduce

batch_size

slightly and add more replicas instead. This setup should eliminate the 15+ minute lag you were seeing and keep latency close to seconds, even at high throughput. If you’d like, share your approximate task-run event rate and current recorder/persister settings, and I can suggest precise target values for your environment.

High level: Prefect uses Docket to coordinate background “perpetual” services. It’s not a classic single “leader process” with heartbeats; instead, Docket uses Redis-backed locks and a task queue to ensure only one instance executes a given perpetual task at a time, while letting you run many API pods across clusters. How it keeps one active executor per singleton service - Perpetual tasks are scheduled into a Redis-backed queue with a stable key (e.g., “scheduler”, “mark-late-runs”, etc.). When you call docket.add(func, key=...), that key de-duplicates scheduling across all pods: every pod can try to schedule, but only one keyed task is active. - Workers consume tasks from a Redis Stream using a consumer group. Exactly one worker claims and runs a given message at a time. - Workers send heartbeats to Redis (about every 2s). If a worker dies, its message lease goes idle. After a redelivery timeout (~5 minutes by default), another worker auto-claims and continues the task. That’s the failover path. - There isn’t a global “leader election”; it’s cooperative single-execution per task key enforced by Redis locks/streams and message claims. The result is functionally one active executor per singleton service. Key mechanics to know - Redis keys/queues - Stream: {docket_name}:stream - Scheduled queue (ZSET): {docket_name}:queue - Consumer group: {docket_name} - Workers set/heartbeats: {docket_name}:workers - Optional lock used only when auto-scheduling perpetual tasks - Heartbeats and TTL - Worker heartbeats every ~2s; workers considered dead after ~10s of missed heartbeats (config inside Docket). - Execution leases are renewed periodically; if a worker dies, messages are re-delivered after the redelivery timeout (~5 minutes) via XAUTOCLAIM. - No fencing tokens - Docket relies on lease renewal and Redis stream semantics, not fencing tokens. Duplicate execution is avoided by keyed scheduling and single-claim processing, but a brief overlap is theoretically possible during crash/recovery windows. - Perpetual integration in Prefect - Prefect registers each singleton service with a stable key via docket.add(..., key="service-name") so any number of API pods can start services; only one pod will actively process the keyed task at a time. - Services run at a defined interval (“every” N seconds). Re-scheduling reuses the same key to avoid duplicates. Operational caveats in multi-cluster - NTP/clock sync: Required. Significant skew can cause premature/late worker eviction and odd failovers. - Network jitter/partitions: If one cluster has intermittent Redis connectivity, you may see task “flapping” (ownership moves after redelivery timeout). - Redelivery timeout trade-off: Default ~5 minutes means worst-case failover time is up to 5 minutes if a worker hard-crashes mid-task. You can lower it if you want faster failover at the cost of more churn. - Stable keys: Ensure Prefect’s built-in services use their default stable keys; don’t override with per-pod-unique keys or you’ll get duplicate execution. - Shared Docket settings: All pods/clusters must use the exact same Docket URL and name so they operate on the same Redis keys and consumer group. What you should see - Many pods start the same service, but only one will actively execute each keyed perpetual task at any given time. - If the active pod dies, another pod will pick up that service’s execution after the redelivery window. - Consumer-group services (TaskRunRecorder, ReactiveTriggers, Actions) scale linearly — all pods process different messages concurrently. If you want, I can suggest safe values for the redelivery/heartbeat intervals and a checklist to validate you’re seeing a single active executor per service in logs.

Here’s a safe, practical tuning guide and a validation checklist. What you can and cannot tune today - Exposed by Prefect settings: - Docket name and backend: - PREFECT_SERVER_DOCKET_NAME (default: prefect-server) - PREFECT_SERVER_DOCKET_URL (default: memory://; set to redis://… for HA) - Not exposed by Prefect (come from pydocket defaults): - Heartbeat interval: ~2s - Missed heartbeats before removal: ~5 (worker considered dead after ~10s) - Redelivery timeout (lease idle time before a task is reclaimed): ~5 minutes - Worker reconnection delay on Redis errors: ~5s Given these are not configurable via Prefect env vars in 3.6.24, the best you can do without patching is to design around them. Safe “targets” to keep in mind - Effective liveness window: ~10 seconds (2s heartbeat x 5 misses) - Worst-case failover for a crashed singleton task: up to ~5 minutes (redelivery timeout) - Practical guidance: - Ensure NTP time sync across clusters to avoid heartbeat misjudgments. - Keep network latency/jitter to Redis low and stable (aim for <50ms p95). - Prefer running the Redis Docket backend close to your API pods. Validation checklist: confirm there’s exactly one active executor per singleton service 1) Confirm shared Docket - On multiple API pods: -

prefect config view | grep -E "PREFECT_SERVER_DOCKET_URL|PREFECT_SERVER_DOCKET_NAME"

- Values must be byte-for-byte identical. 2) Verify services are enabled -

prefect server services ls

- Ensure your singleton services (e.g., Scheduler, MarkLateRuns, CancellationCleanup, ProactiveTriggers, etc.) show enabled. 3) Observe service scheduling logs - Look for logs like: - “Registered perpetual service … key=…” - “Scheduled perpetual task … key=… next run in …” - You should see these on multiple pods at startup, but only one pod should then show actual execution logs repeatedly. 4) Observe execution logs for a singleton service - Pick one service (e.g., Scheduler). In logs across pods/clusters: - Only one pod should repeatedly emit messages like “scheduler: starting cycle”, “scheduler: completed cycle (X runs)”. - Other pods should be mostly quiet for that service (aside from occasional scheduling/no-op messages). 5) Failover test - Identify the pod that is actively executing the service (e.g., Scheduler). - Kill that pod. - Expectation: - Within ~10s, its worker heartbeat disappears. - Within the redelivery window (up to ~5 min), another pod begins executing that service (you’ll see the same “starting cycle” logs appear on a different pod). - No overlapping cycles should occur during steady state; a small overlap during failover is possible right at takeover boundaries. 6) Watch Redis state (optional) - Check the worker set in Redis: - ZSET {docket_name}:workers should contain your active workers with recent timestamps. - During failover, the previous worker’s entry ages out (~10s) and a new worker starts claiming tasks. 7) Look for signs of flapping - Repeated “starting cycle” logs rapidly bouncing between pods without kills typically indicates network instability to Redis or clock skew. - Action: stabilize network, ensure time sync, or reduce cross-region latency. If you really need different timeouts - Today, Prefect does not expose Docket’s heartbeat/redelivery as settings. Changing them requires: - Editing Prefect’s server code to instantiate the Docket Worker with custom parameters, or - Opening a feature request to expose these via PREFECT_SERVER_DOCKET_* settings. - If you want help crafting a minimal patch to set a shorter redelivery timeout (e.g., 60–120s), I can provide a snippet and the exact file/lines to modify. Given your multi-cluster setup and heavy load, I’d focus on: - Keeping Redis close and stable - Scaling consumer-group services (TaskRunRecorder, EventPersister) horizontally - Verifying singleton services show one active executor at a time with the checklist above

- Running a controlled failover test to confirm takeover timing matches expectations

Short answer: No — do not use a different PREFECT_SERVER_DOCKET_NAME for API and background services. They should be identical across all API/background-service pods and clusters. Why the name must be the same - Docket name is the namespace for Redis keys (streams, locks, worker sets). If you change the name, you create a separate “island” of coordination. - Singleton/leader-elected services (Scheduler, MarkLateRuns, CancellationCleanup, etc.) rely on Docket to ensure only one active executor. Different names = no coordination = duplicate execution, race conditions, and potential DB deadlocks. - Running API and services with different names while pointing at the same Postgres will cause “split‑brain” behavior. Use different names only when - You are intentionally running completely isolated Prefect server stacks (separate databases and message brokers) that must not coordinate. In that case, different names are fine. Do not point different Docket names at the same Postgres. Recommended configuration

# All API and background service pods in all clusters:
PREFECT_SERVER_DOCKET_URL=redis://<host>:6379/<db>
PREFECT_SERVER_DOCKET_NAME=prefect-server   # same everywhere

If you need to separate concerns (e.g., run services in one cluster and only API/UI in another), keep the same Docket name and simply disable singleton services on the cluster that should not run them:

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=false
PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=false
PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=none
# (and any other singleton services you’ve enabled)

Summary - Same Docket URL + same Docket name across all instances = safe, coordinated single active executor per singleton service. - Different names against the same DB = split‑brain and the kinds of deadlocks/races you’ve observed.

The Repossessor is a background service that reconciles expired concurrency leases and frees up their slots. What it does - Finds expired concurrency leases in the shared lease storage (typically Redis). - For each expired lease, it: - Decrements the active slot counts on the associated concurrency limits in Postgres. - Updates occupancy metrics (e.g., avg_slot_occupancy_seconds, denied_slots decay). - Removes the lease record from the lease storage. Why it matters - Prevents “stuck” concurrency slots when a task/flow ends unexpectedly or a lease isn’t explicitly released. - Keeps concurrency limits accurate so new work isn’t blocked by orphaned/expired leases. - Maintains realistic occupancy/denied metrics for decay-based limiting. Where it operates - Lease storage: Redis (or other configured storage) holding ResourceLease records with expiration, resource_ids (concurrency limit IDs), and metadata (slots held, holder info). - Database: ConcurrencyLimitV2 table — adjusts active_slots and metrics; updates timestamps for decay calculations. How it runs - A perpetual service that scans for expired leases on an interval (default ~15s). - Queues per-lease revocations via Docket; each revocation task safely updates DB state and removes the lease. When to enable it - In production: keep it enabled (default is enabled). Disabling it can leave expired leases uncollected, causing artificial saturation of your concurrency limits. Caveats - For multi-cluster: ensure all API instances use the same Redis-backed lease storage and the same Postgres writer so repossession is consistent. If any instance uses in-memory lease storage, do not run multiple Repossessors. - If you see deadlocks, it’s usually due to split-brain (different Docket names/URLs) or cross-cluster DB contention — not the Repossessor logic itself. If you want, I can share the exact env settings to control its loop interval or help you verify it’s running and reclaiming slots correctly in your logs.

Here’s a sane, HA-friendly configuration for Redis messaging and Docket that works across multiple clusters. Recommended (separate logical DBs for isolation)

# Use Redis as the messaging broker
PREFECT_MESSAGING_BROKER=prefect_redis.messaging

# Prefer a full URL; it takes precedence over host/port/db vars
PREFECT_REDIS_MESSAGING_URL=redis://<redis-host>:6379/0

# If you’re not using the URL, these are honored:
PREFECT_REDIS_MESSAGING_HOST=<redis-host>
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0

# Docket (leader coordination for singleton services)
PREFECT_SERVER_DOCKET_URL=redis://<redis-host>:6379/1
PREFECT_SERVER_DOCKET_NAME=prefect-server   # same across all pods/clusters

Notes and caveats - URL precedence: If

PREFECT_REDIS_MESSAGING_URL

is set, it overrides

PREFECT_REDIS_MESSAGING_HOST/PORT/DB

. Use the URL form to avoid ambiguity. - Separate DBs: Using

/0

for messaging and

/1

for Docket keeps keys and monitoring clean. It’s not required, but strongly recommended. - Managed Redis cluster mode: Some providers only allow DB 0. In that case, set: -

PREFECT_REDIS_MESSAGING_URL=redis://<host>:6379/0

-

PREFECT_SERVER_DOCKET_URL=redis://<host>:6379/0

This is safe because Prefect/Docket namespace their keys; just ensure the same

PREFECT_SERVER_DOCKET_NAME

across all pods so singleton services coordinate correctly. - TLS/auth: Use rediss and embed credentials in the URL if needed:

PREFECT_REDIS_MESSAGING_URL=rediss://:<password>@<host>:6379/0
  PREFECT_SERVER_DOCKET_URL=rediss://:<password>@<host>:6379/1

- Multiple clusters: All API/background-service pods in all clusters must use identical values for

PREFECT_REDIS_MESSAGING_URL

,

PREFECT_SERVER_DOCKET_URL

, and

PREFECT_SERVER_DOCKET_NAME

. Validation - On a pod in each cluster, confirm:

prefect config view | grep -E "PREFECT_MESSAGING_BROKER|PREFECT_REDIS_MESSAGING_URL|PREFECT_REDIS_MESSAGING_DB|PREFECT_SERVER_DOCKET_URL|PREFECT_SERVER_DOCKET_NAME"

- Watch logs on startup for successful connections to Redis (messaging and Docket) and ensure you see only one active executor for singleton services (e.g., Scheduler) across the fleet. If you share your Redis provider (self-hosted vs ElastiCache/Azure/Redis Enterprise) I can tailor the exact URL syntax and any provider-specific gotchas.

Short answer: It’s a server background service. You can run it on API pods (default) or in a separate “services-only” deployment — just don’t run it on workers. Two supported patterns - Co-located on API pods (simple/default) - Leave it enabled everywhere you run the API; Docket will ensure only one executor is active at a time. - Pros: simplest HA setup. Cons: shares CPU with API. - Dedicated services deployment - Run a separate Deployment that starts only background services (e.g.,

prefect server services start

) and disable services on API pods. - Pros: isolates service load. Cons: extra component to manage. Environment toggles - Enable/disable: -

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true|false

- Scan interval: -

PREFECT_SERVER_SERVICES_REPOSSESSOR_LOOP_SECONDS=15

(example) - Critical shared backends (same across all pods/clusters): -

PREFECT_SERVER_DOCKET_URL=redis://<host>:6379/1

-

PREFECT_SERVER_DOCKET_NAME=prefect-server

- Messaging/DB as you’ve already configured Examples - API pods with repossessor enabled (co-located):

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true

- Dedicated services pod only: - Services pod:

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true

- API pods:

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=false

Verification - Logs should show on exactly one pod at a time: - “monitor_expired_leases loop started” - “revoking expired lease …” - If you kill the active pod, another pod should begin logging those messages within the next loop or two. Bottom line: Run it wherever you run Prefect Server services. By default, that’s on API pods; if you prefer separation, run it in a services-only deployment and disable it on the API pods. In all cases, keep the same Redis-backed Docket URL + name so only one executor is active.

@Marvin - Using prefect version-3.6.24, observed an warning prefect.server.utilities.user_templates - unhandled exception rendering template. TypeError: object of type UUID is not JSON serializable

@Marvin - Using prefect version-3.6.24, observed an warning prefect.server.utilities.user_templates - unhandled exception rendering template. TypeError: object of type UUID is not JSON serializable

Thanks for the details. That warning comes from the Jinja-based user template renderer in Prefect Server when a template tries to JSON-encode a UUID. What’s happening - Many objects in the template context (automation, event, firing, flow_run, deployment, task_run, etc.) have

.id

fields that are Python UUID objects. - If a template uses the Jinja

tojson

filter on a UUID, Python’s json module raises: TypeError: object of type UUID is not JSON serializable - You’ll see a warning like: prefect.server.utilities.user_templates - unhandled exception rendering template. TypeError: object of type UUID is not JSON serializable Where you typically hit this - Webhook payload templates (CallWebhook action) - Notification templates - RunDeployment parameter templates How to fix templates safely - Prefer direct interpolation or convert to string before JSON encoding: - Direct (Jinja auto-coerces to string):

{{ flow_run.id }}

- Explicit string conversion:

{{ flow_run.id | string }}

- If you’re building JSON yourself, only apply tojson to fields that are already JSON-serializable, and stringify UUIDs:

{
      "flow_run_id": "{{ flow_run.id | string }}",
      "automation_id": "{{ automation.id | string }}",
      "name": {{ deployment.name | tojson }}
    }

- Avoid: -

{{ flow_run.id | tojson }}

# triggers the UUID serialization error Tips - If you need a pure-JSON payload, stringify all UUIDs in the template. - For debugging, temporarily remove

tojson

from expressions to find which value contains a UUID. - After rendering, Prefect may try to coerce a rendered scalar or object back to native JSON when appropriate, but the tojson step happens first — so keep UUIDs out of tojson. If you can share the specific template you’re using, I can mark exactly which fields to wrap with

| string

and which can safely use

| tojson

.

Thanks for the detail. That warning comes from Prefect’s sandboxed Jinja renderer for user-defined templates (used by actions like SendNotification, CallWebhook, RunDeployment). In 3.6.24, UUIDs in the template context can trigger a TypeError when passed through Jinja’s tojson filter, since Python’s json module doesn’t serialize UUID by default. What’s happening - Module: prefect.server.utilities.user_templates - It renders Jinja templates with a safe environment. - Some fields are auto-wrapped with | tojson for type preservation. - If the context contains a UUID (e.g., flow_run.id, deployment.id, task_run.id), tojson -> json.dumps(...) raises “object of type UUID is not JSON serializable”. Where you’ll see it - Actions/templates that include: - SendNotification subject/body - CallWebhook payload - RunDeployment parameters (auto-wrapped via tojson) - Any template using {{ some_uuid | tojson }} or auto-wrapped content that includes UUIDs. Workarounds you can apply today - Avoid tojson on UUID values; coerce to string: - Use "{{ flow_run.id }}" instead of "{{ flow_run.id | tojson }}" - Or explicitly: "{{ flow_run.id | string | tojson }}" - If you need JSON for a bigger object that includes UUIDs, cast UUID fields to string in the template: - Example:

{
      "flow_run_id": "{{ flow_run.id }}",
      "count": {{ count | tojson }}
    }

- As an advanced workaround, register a custom tojson that handles UUIDs (default=str). If you control the server process and can extend filters:

from prefect.server.utilities.user_templates import register_user_template_filters
  import json

  def tojson_uuid_safe(obj):
      return json.dumps(obj, default=str)

  register_user_template_filters({"tojson": tojson_uuid_safe})

Note: this is process-local; make sure all API pods load the filter if you go this route. Recommended next steps - Update any notification/webhook/automation templates that reference IDs: - Replace

| tojson

with plain string interpolation for UUIDs. - Keep

| tojson

for numeric/dict data where needed. - If you hit this in RunDeployment parameter templating, ensure parameter values that may be UUIDs are rendered as strings in the template. If you can share a small snippet of the template that triggers the warning, I can suggest the minimal change to fix it while preserving your intended output.

Thanks for the context — this is a known risk when multiple Task Run Recorder instances write concurrently under load: concurrent upserts into task_runs/task_run_states can deadlock if two transactions touch overlapping rows in different orders. What’s likely happening - Multiple recorder pods (across clusters) are flushing batches that include events for some of the same task_run ids or natural keys. Postgres can form a cycle when each transaction locks rows in a different order, leading to DeadlockDetectedError on commit/flush. Immediate mitigations (pick 1–2 to confirm the cause) - Temporarily run a single Task Run Recorder group - Scale down to one API pod with recorder enabled (or enable recorder in only one cluster) to verify deadlocks stop. If they do, it’s concurrency on the same rows causing the issue. - Reduce cross-pod overlap by tuning batching - Start with:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=50
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=20
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1

- Smaller DB batch_size reduces the number of rows locked at once (and the lock window), which usually lowers deadlock frequency. Keep read_batch_size moderate so you still drain the queue efficiently. - Keep consumer-group uniqueness - Ensure all recorder pods are in the same consumer group (this is the default) and that there is only one Redis broker (no shadow brokers). If any recorder points at a different broker/db, the same events can be processed by multiple pods and collide in Postgres. Database-side visibility and hygiene - Turn on deadlock diagnostics so we can see the exact relations involved:

ALTER SYSTEM SET deadlock_timeout = '2s';
  ALTER SYSTEM SET log_lock_waits = on;
  SELECT pg_reload_conf();

Then share the deadlock report from Postgres logs; it will show which tables/rows were in the cycle (typically task_runs and task_run_states). - Check for FK/unique constraints that can increase lock scope - List constraints:

SELECT conname, conrelid::regclass AS table, confrelid::regclass AS ref_table, contype
    FROM pg_constraint
    WHERE conrelid::regclass::text IN ('task_runs','task_run_states');

- If there’s a FK from task_runs.state_id → task_run_states.id, it can participate in lock inversions during concurrent upserts. If present, consider removing that FK and relying on application-level guarantees — it has been a source of deadlocks in high-throughput paths. - Monitor lock contention:

SELECT bl.pid AS blocked_pid, ka.query AS blocking_query, now() - ka.query_start AS blocking_for,
         kl.relation::regclass AS blocking_relation, a.query AS blocked_query
  FROM pg_locks bl
  JOIN pg_stat_activity a ON a.pid = bl.pid
  JOIN pg_locks kl ON kl.locktype = bl.locktype AND kl.relation = bl.relation AND kl.granted
  JOIN pg_stat_activity ka ON ka.pid = kl.pid
  WHERE NOT bl.granted;

Throughput vs safety tuning (pick a lane) - If your main pain is deadlocks: - Lower batch_size (e.g., 10–20) and reduce the number of recorder replicas to 1–2. This shortens transactions and lowers cross-pod overlap. - If your main pain is lag: - Increase replicas but keep batch_size modest (20–50). Avoid very large batches; large multi-row upserts across many keys raise deadlock probability. Operational tips - Keep all recorder instances physically close (network-wise) to the same Postgres to minimize clock/latency skew. - Ensure PgBouncer (if used) is in session pooling mode for the recorder connections and has enough pool size to avoid stalls under bursts. - Verify all API/recorder pods point to the writer endpoint (not a reader or mixed R/W proxy). Upgrade path - There have been improvements to the recorder’s bulk write path and deadlock resilience in newer 3.x releases. If you can, test on the latest 3.x — several teams have seen fewer (or no) deadlocks after upgrading.

What would help me give exact guidance - The Postgres deadlock report snippet (it names the relations and queries in the cycle) - Your current recorder settings (read_batch_size, batch_size, flush_interval), number of recorder pods, and approximate task-run events/sec - Whether task_run.state_id has an FK to task_run_states.id in your DB Short-term “get stable now” option - Run Task Run Recorder only in one cluster (or even a single pod) while you tune batch sizes and gather deadlock logs. Once stable, scale back up gradually (add 1 pod at a time) and keep batch_size in a conservative range (20–50).

Thanks for sharing your current Task Run Recorder config. With one recorder pod per cluster and batch sizes of 50, you’ve got good throughput — but concurrent upserts from multiple recorders can still deadlock when they touch overlapping task_runs/state rows at the same time. Practical mitigations to try now - Reduce DB batch size, keep broker read size - Keeps transactions shorter and reduces lock overlap while still draining the queue quickly. - Suggested:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=100
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=20
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1

- Stagger flush timings across clusters - Avoid synchronized commits by slightly offsetting intervals: - Cluster A: FLUSH_INTERVAL=1.00 - Cluster B: FLUSH_INTERVAL=1.15 (or 1.25) - Confirm single consumer group and single broker - All recorder pods must use the same Redis broker and the same consumer group (default for the recorder). Any stray recorder pointing at a different broker/DB will reprocess the same events and increase collisions. - As a baseline test, run a single global recorder - Temporarily disable the recorder in one cluster to confirm deadlocks stop. If they do, the issue is row-level overlap between recorders — then keep batch_size smaller and add back the second recorder. Postgres visibility and tuning - Enable deadlock diagnostics (helps pinpoint exact tables/queries):

ALTER SYSTEM SET deadlock_timeout = '2s';
  ALTER SYSTEM SET log_lock_waits = on;
  SELECT pg_reload_conf();

- Capture the deadlock report from Postgres logs and share it; it will name the relations (typically task_runs and task_run_states) and lock order. - Sanity checks: - Ensure the recorder connects to the writer/primary endpoint (not a reader or mixed proxy). - If you use PgBouncer, prefer session pooling for recorder connections and raise pool size so commits don’t queue behind long-lived sessions. If deadlocks persist - Lower DB batch_size further (e.g., 10–15) and/or reduce the number of recorder pods (1 global recorder is the simplest way to eliminate cross-pod overlap). - Keep READ_BATCH_SIZE high (50–100) so you still read efficiently from Redis, but write to Postgres in smaller chunks. Helpful SQL to inspect constraints (to see what’s involved in the deadlock)

-- List relevant constraints
SELECT conname, conrelid::regclass AS table, confrelid::regclass AS ref_table, contype
FROM pg_constraint
WHERE conrelid::regclass::text IN ('task_runs','task_run_states');

-- Observe blocking/blocked locks when it happens
SELECT bl.pid AS blocked_pid, ka.query AS blocking_query, now() - ka.query_start AS blocking_for,
       kl.relation::regclass AS blocking_relation, a.query AS blocked_query
FROM pg_locks bl
JOIN pg_stat_activity a ON a.pid = bl.pid
JOIN pg_locks kl ON kl.locktype = bl.locktype AND kl.relation = bl.relation AND kl.granted
JOIN pg_stat_activity ka ON ka.pid = kl.pid
WHERE NOT bl.granted;

Next steps - Try the (100, 20, 1) settings and add a 10–20% flush interval offset between clusters. - If you still see DeadlockDetectedError, please share one deadlock report from the Postgres logs plus your recorder replica counts. I can then give you a precise plan (e.g., batch sizing, replica counts, or consolidating to a single global recorder) to get you stable.

Great question. The Task Run Recorder uses a Redis Streams consumer group. You can verify that all pods are in the same group and see their membership and lag directly in Redis. What to expect - Group name: task-run-recorder - Topic/stream: the “events” topic (actual Redis stream key name depends on the broker’s key prefix) - Each recorder pod joins the same group with a unique consumer name How to check with Redis CLI 1) Connect to the same Redis DB used for messaging (not the Docket DB) - Use the exact URL from PREFECT_REDIS_MESSAGING_URL (or host/port/db if you’re not using the URL form)

redis-cli -u "$PREFECT_REDIS_MESSAGING_URL" ping

2) Find the stream key(s) - If you know the key, skip to step 3. Otherwise, scan for streams in the DB and inspect them:

# List all stream keys (Redis >= 6)
redis-cli -u "$PREFECT_REDIS_MESSAGING_URL" --raw SCAN 0 TYPE stream

# If TYPE is not available, fallback:
redis-cli -u "$PREFECT_REDIS_MESSAGING_URL" --raw SCAN 0 MATCH *events*
# Then for each found key:
redis-cli -u "$PREFECT_REDIS_MESSAGING_URL" TYPE <key>

3) List consumer groups on each candidate stream - Look for a group named task-run-recorder

redis-cli -u "$PREFECT_REDIS_MESSAGING_URL" XINFO GROUPS <stream_key>

4) Inspect the consumers in the group (one per recorder pod)

redis-cli -u "$PREFECT_REDIS_MESSAGING_URL" XINFO CONSUMERS <stream_key> task-run-recorder

- You should see one consumer entry per recorder pod (names are unique per pod). Check idle time and pending counts. 5) Check pending and lag for the group

# Pending messages summary for the group
redis-cli -u "$PREFECT_REDIS_MESSAGING_URL" XPENDING <stream_key> task-run-recorder

# (Redis 7+) Group info may include 'lag' in XINFO GROUPS output

Tips - Make sure you connect to the same Redis DB index you configured in PREFECT_REDIS_MESSAGING_URL (e.g., redis://host:6379/0). If you separated Docket to a different DB (e.g., /1), use the messaging DB here. - Enable DEBUG logs on API pods to see the recorder’s startup lines; they typically include the topic (“events”), the group (“task-run-recorder”), and the consumer name it registered with. - If you don’t see task-run-recorder in any stream’s groups, double‑check: - PREFECT_MESSAGING_BROKER=prefect_redis.messaging - PREFECT_REDIS_MESSAGING_URL points to the right Redis instance/DB - The recorder service is enabled and running: PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true If you share the output of: - SCAN for stream keys - XINFO GROUPS for the candidate stream(s) …I can help you pinpoint the exact stream key Prefect is using and confirm all recorder pods are in the same group.

Great — that separation looks good: messaging on DB 0 and Docket on DB 1. Here’s how to verify the Task Run Recorder consumer group on your messaging Redis (DB 0): 1) Sanity check connectivity to the messaging DB (DB 0)

redis-cli -u "<redis://host:6379/0>" ping

2) Find the stream key(s) used for events - If your Redis supports TYPE with SCAN:

redis-cli -u "<redis://host:6379/0>" --raw SCAN 0 TYPE stream

- If TYPE is not available, scan and filter by name, then confirm TYPE:

# Find likely event streams (names vary by setup)
redis-cli -u "<redis://host:6379/0>" --raw SCAN 0 MATCH "*events*"
# For each candidate key:
redis-cli -u "<redis://host:6379/0>" TYPE <stream_key>

3) List consumer groups on each candidate stream - You’re looking for a group named task-run-recorder

redis-cli -u "<redis://host:6379/0>" XINFO GROUPS <stream_key>

4) Inspect the recorder consumers in that group - Expect one consumer entry per recorder pod (each pod has a unique consumer name)

redis-cli -u "<redis://host:6379/0>" XINFO CONSUMERS <stream_key> task-run-recorder

- Check: - name: unique per pod - pending: how many messages are assigned to that consumer - idle: how long it’s been since the consumer last acknowledged a message 5) Check pending and lag

# Pending overview for the group
redis-cli -u "<redis://host:6379/0>" XPENDING <stream_key> task-run-recorder

# Stream length (rough volume)
redis-cli -u "<redis://host:6379/0>" XLEN <stream_key>

Optional: one-liner to scan all streams and print any groups named task-run-recorder

for k in $(redis-cli -u "<redis://host:6379/0>" --raw SCAN 0 TYPE stream | awk 'NR>2'); do
  if redis-cli -u "<redis://host:6379/0>" XINFO GROUPS "$k" 2>/dev/null | grep -q "task-run-recorder"; then
    echo "Stream: $k"
    redis-cli -u "<redis://host:6379/0>" XINFO GROUPS "$k"
    redis-cli -u "<redis://host:6379/0>" XINFO CONSUMERS "$k" task-run-recorder
  fi
done

If you don’t see task-run-recorder anywhere: - Confirm on the pods: -

prefect config view | grep -E "PREFECT_MESSAGING_BROKER|PREFECT_REDIS_MESSAGING_URL"

- Ensure

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

- Check API logs (set PREFECT_LOG_LEVEL=DEBUG) for lines indicating the recorder started; it typically logs the group (“task-run-recorder”), stream key, and its unique consumer name. - Make sure you’re connected to DB 0 (PREFECT_REDIS_MESSAGING_URL=redis://host:6379/0), not DB 1 (that’s only for Docket). If you share the XINFO output for the stream key you find, I can help verify that both clusters’ recorder pods are in the same group and reading as expected.

You can tail the Redis Streams directly to see the events your API server is publishing. Since you’re using PREFECT_REDIS_MESSAGING_URL=redis://host:6379/0 and Docket on /1, do all of the following against DB 0. Step-by-step (safe, read-only) 1) Connect to the messaging DB (DB 0)

redis-cli -u "<redis://host:6379/0>" ping

2) Discover the stream key(s) - Prefect uses Redis Streams for messaging; the exact key name can vary. Scan for streams, then narrow down by name:

redis-cli -u "<redis://host:6379/0>" --raw SCAN 0 TYPE stream
# If TYPE is not supported on your Redis:
redis-cli -u "<redis://host:6379/0>" --raw SCAN 0 MATCH "*events*"
redis-cli -u "<redis://host:6379/0>" --raw SCAN 0 MATCH "*prefect*"
# For each candidate key, confirm it is a stream:
redis-cli -u "<redis://host:6379/0>" TYPE <stream_key>

3) Inspect stream info and recent entries

# Stream metadata (length, last-generated-id, groups)
redis-cli -u "<redis://host:6379/0>" XINFO STREAM <stream_key>

# Peek at recent messages (adjust COUNT as needed)
redis-cli -u "<redis://host:6379/0>" XRANGE <stream_key> - + COUNT 5

- You’ll see entries like: - ID: 1712861234567-0 - Fields: key/value pairs. One of the fields typically contains the event payload (JSON). Use the above XRANGE output to identify the exact field name (often something like data/payload/message). 4) “Tail -f” new events without consuming them - This listens for new messages from “now” and does not affect consumer groups or pending lists:

redis-cli -u "<redis://host:6379/0>" XREAD BLOCK 0 STREAMS <stream_key> $

- Leave this running to see events live as the API publishes them. Optional: check the Task Run Recorder consumer group - Group name is typically task-run-recorder. To confirm membership and lag:

redis-cli -u "<redis://host:6379/0>" XINFO GROUPS <stream_key>
redis-cli -u "<redis://host:6379/0>" XINFO CONSUMERS <stream_key> task-run-recorder
redis-cli -u "<redis://host:6379/0>" XPENDING <stream_key> task-run-recorder

Python snippet to pretty-print payloads - If you want structured logging and JSON parsing:

import json, time
import redis

STREAM = "<stream_key>"
r = redis.Redis.from_url("<redis://host:6379/0>")
last_id = "$"  # start from new messages; use "0-0" to read backlog

while True:
    resp = r.xread({STREAM: last_id}, block=10000, count=100)
    if not resp:
        continue
    for key, messages in resp:
        for mid, fields in messages:
            # Inspect which field holds the JSON payload
            payload = fields.get(b"data") or fields.get(b"payload") or fields.get(b"message")
            if payload:
                try:
                    evt = json.loads(payload)
                    print(f"{mid} event={evt.get('event')} resource={evt.get('resource')}")
                except Exception:
                    print(f"{mid} raw payload: {payload!r}")
            else:
                print(f"{mid} fields: {fields}")
            last_id = mid

Important cautions - Do not use XREADGROUP in production just to “watch” messages — groups/consumers will claim messages and can impact your services. Use XREAD with a plain stream instead. - MONITOR will show every Redis command but is extremely noisy and can impact performance; only use it briefly if you must:

redis-cli -u "<redis://host:6379/0>" MONITOR | grep XADD

If you paste the XRANGE output for your candidate stream, I can help you identify the exact field containing the event JSON and tailor a jq or Python formatter for your setup.

PSUBSCRIBE won’t show Prefect events. Prefect uses Redis Streams (XADD/XREAD), not Pub/Sub (PUBLISH/PSUBSCRIBE). That’s why you’re not seeing messages with PSUBSCRIBE. Use XINFO/XRANGE/XREAD instead. Here are copy/paste commands that work against your messaging DB (db 0): If you have a password - One-off container with redis-cli:

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" ping

Find the stream key(s) - Try TYPE-filtered scan (Redis 7+):

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" --raw SCAN 0 TYPE stream

- Fallback (any Redis), then confirm each key’s type:

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" --raw SCAN 0 MATCH "*events*"

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" TYPE <stream_key>

Inspect the stream and groups

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" XINFO STREAM <stream_key>

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" XINFO GROUPS <stream_key>

- You should see a group named

task-run-recorder

(and possibly others). Tail new events (non-destructive)

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" XREAD BLOCK 0 STREAMS <stream_key> $

- Leave this running to see new events as they’re published. One of the fields (often

data

or

payload

) will contain the JSON event. Peek at recent messages

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" XRANGE <stream_key> - + COUNT 5

If you prefer exec’ing into a running container - Make sure you exec into a container that actually has redis-cli (typically the Redis container, not the Prefect API container). For example:

docker exec -it <redis-container-name> redis-cli -a <PASSWORD> -n 0 XINFO STREAM <stream_key>

Optional: verify XADD operations (noisy, use briefly)

docker run --rm -it redis:7-alpine \
  redis-cli -u "redis://:<PASSWORD>@host:6379/0" MONITOR | grep XADD

Common gotchas - PSUBSCRIBE subscribes to Pub/Sub channels; it does not show Stream entries. - Ensure you connect to DB 0 (messaging) — your Docket is on DB 1. - If you don’t see any groups with XINFO GROUPS, double-check you found the correct stream key or that the recorder is running/enabled. If you paste the XRANGE output for the stream you find, I can help identify which field contains the JSON and suggest a jq/grep to pretty-print it.

Use the stream named “events”. That’s the Redis Stream where the API publishes Prefect events, and it’s what the background services (Event Persister, Task Run Recorder, Reactive Triggers, etc.) consume. Quick commands against your messaging Redis (DB 0) - Stream metadata

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO STREAM events

- See consumer groups on the events stream

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO GROUPS events

- Inspect consumers in a specific group (e.g., task-run-recorder)

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO CONSUMERS events task-run-recorder

- Tail new events non-destructively (like tail -f)

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XREAD BLOCK 0 STREAMS events $

- Peek at recent messages

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XRANGE events - + COUNT 5

What you’ll see in messages - Each entry has fields; one typically contains the JSON event payload (often under “data” and a small metadata blob under “attributes”). Use XRANGE to see exact field names for your build, then parse accordingly. If XINFO STREAM events errors - Make sure you’re connected to DB 0 (messaging), not DB 1 (Docket). - If the stream doesn’t exist yet, trigger a small flow/task to generate events, then re-run the command. - To discover streams:

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 --raw SCAN 0 TYPE stream

Got it. There are two good ways to “stream” Prefect events, depending on what you mean by “in redis logs”: A) From Redis itself (tail the events stream) Prefect publishes events to a Redis Stream named

events

(DB 0 in your setup). You can non-destructively watch them with XREAD: - Tail new events (like tail -f):

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XREAD BLOCK 0 STREAMS events $

- Peek at recent messages (to learn field names like data/attributes):

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XRANGE events - + COUNT 5

- See consumer groups (to verify services consuming events):

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO GROUPS events

Note: Redis Streams are not Pub/Sub; PSUBSCRIBE won’t show these messages. B) From Prefect (enable DEBUG logs for the messaging pipeline) If you want the Prefect API to log the messaging activity to its logs, turn on DEBUG for the messaging loggers: - Minimal:

export PREFECT_LOG_LEVEL=DEBUG
export PREFECT_LOGGING_EXTRA_LOGGERS="prefect.server.events.messaging,prefect_redis.messaging,prefect.server.events.services.event_persister,prefect.server.services.task_run_recorder"

- Or with a full logging settings override (JSON) to force DEBUG for those specific loggers:

export PREFECT_LOGGING_SETTINGS='{
  "loggers": {
    "prefect.server.events.messaging": {"level": "DEBUG"},
    "prefect_redis.messaging": {"level": "DEBUG"},
    "prefect.server.events.services.event_persister": {"level": "DEBUG"},
    "prefect.server.services.task_run_recorder": {"level": "DEBUG"}
  }
}'

Then restart your API pods. You’ll see detailed logs about publishing/consuming batches, flushes, and any errors. If you literally want messages in the Redis server logs Redis doesn’t log stream payloads to its server log. You can approximate this with MONITOR (very noisy; not recommended in prod) to see XADD operations:

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 MONITOR | grep XADD

But for actual event contents, use XRANGE/XREAD as above. If you want help formatting the event JSON from XRANGE/XREAD, paste a sample entry and I’ll show you a jq/awk one-liner to pretty-print just the event type and resource.

XINFO STREAM shows metadata only. To see just the event payload fields from the stream entries, use XRANGE/XREAD and filter for data and attributes. One-shot: print last 5 entries’ data/attributes

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XRANGE events - + COUNT 5 |
  awk '"'"'
    /^[0-9-]+$/           {print "---"; print "id: "$0; next}
    $0=="data"            {getline v; print "data: " v; next}
    $0=="attributes"      {getline v; print "attributes: " v; next}
  '"'"'
'

Tail live events (non-destructive “follow”)

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ |
  awk '"'"'
    /^[0-9-]+$/           {print "---"; print "id: "$0; next}
    $0=="data"            {getline v; print "data: " v; next}
    $0=="attributes"      {getline v; print "attributes: " v; next}
  '"'"'
'

If your redis-cli supports JSON output (Redis 7+), you can use jq to extract fields precisely:

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --json XRANGE events - + COUNT 5 |
  jq -r ".[] | {id: .[0], obj: (reduce range(0; (.[1]|length); 2) as \$i ({}; . + { (.[1][\$i]): (.[1][\$i+1]) }))} | {id, data: .obj.data, attributes: .obj.attributes}"
'

Notes - These commands assume your event entries store the JSON payload under data and a small metadata blob under attributes. If your fields differ, first run:

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 --raw XRANGE events - + COUNT 1

and adjust the awk/jq keys accordingly. - Use DB 0 for messaging (your Docket is on DB 1). - XREAD (without GROUP) is safe for “tapping” the stream; it won’t claim or acknowledge messages used by your background services.

XINFO STREAM only shows metadata. To print the data field from actual messages, use XRANGE (for recent entries) or XREAD (to “tail”) and filter for the data field. Print last N entries’ data

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XRANGE events - + COUNT 10 |
  awk '"'"'
    /^[0-9-]+$/           {id=$0; next}        # capture message id
    $0=="data"            {getline v; print v} # print only the data value
  '"'"'
'

Tail live data (non-destructive follow)

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ |
  awk '"'"'
    /^[0-9-]+$/           {id=$0; next}
    $0=="data"            {getline v; print v}
  '"'"'
'

If your redis-cli supports JSON output (Redis 7+), you can parse just the data field with jq: Last N entries (JSON-aware)

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --json XRANGE events - + COUNT 10 |
  jq -r ".[] | {id: .[0], fields: (reduce range(0; (.[1]|length); 2) as \$i ({}; . + { (.[1][\$i]): (.[1][\$i+1]) }))} | .fields.data"
'

Notes - These assume the event JSON payload is stored under the field name data. If your entries use a different field (e.g., payload or message), run a quick peek to confirm and adjust:

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 --raw XRANGE events - + COUNT 1

- Use DB 0 (messaging) for events; your Docket is on DB 1. - XREAD (without GROUP) is safe for observing; it does not claim/ack messages from your background services.

Got it. To see “scheduled”, Task Run Recorder, and Event Persister activity, you’ll want two views: - Prefect service logs (best source of what each service is doing) - Optional Redis-level visibility (XADD/XREADGROUP/XAUTOCLAIM activity) A) Turn on detailed Prefect logs for the right components Set these env vars on your API/background-service pods and restart them: Minimal

PREFECT_LOG_LEVEL=DEBUG
PREFECT_LOGGING_EXTRA_LOGGERS="prefect.server.events.messaging,prefect_redis.messaging,prefect.server.events.services.event_persister,prefect.server.services.task_run_recorder,prefect.server.services.scheduler,prefect.server.services.late_runs,prefect.server.services.cancellation_cleanup,prefect.server.services.repossessor,prefect.server.api.background_workers,docket"

Or use a JSON override to force DEBUG per-logger

PREFECT_LOGGING_SETTINGS='{
  "loggers": {
    "prefect.server.events.messaging": {"level": "DEBUG"},
    "prefect_redis.messaging": {"level": "DEBUG"},
    "prefect.server.events.services.event_persister": {"level": "DEBUG"},
    "prefect.server.services.task_run_recorder": {"level": "DEBUG"},
    "prefect.server.services.scheduler": {"level": "DEBUG"},
    "prefect.server.services.late_runs": {"level": "DEBUG"},
    "prefect.server.services.cancellation_cleanup": {"level": "DEBUG"},
    "prefect.server.services.repossessor": {"level": "DEBUG"},
    "prefect.server.api.background_workers": {"level": "DEBUG"},
    "docket": {"level": "DEBUG"}
  }
}'

What you’ll see - Scheduler/“scheduled”: - Acquisition via Docket (look for “acquired lease” / “perpetual service”) - “scheduler cycle start/complete”, “scheduled N flow runs” - Event Persister: - “read batch size”, “flushing N events”, “persisted N events”, queue capacity warnings - Task Run Recorder: - “read N messages”, “flushed N task runs / M states”, and any retry/backoff (e.g., on deadlocks) - Messaging: - Publish/consume details from

prefect.server.events.messaging

and

prefect_redis.messaging

Grab logs from Kubernetes

# Tail all API pods and filter for the services you care about
kubectl logs deploy/<api-deployment> -c <api-container> -f \
  | egrep -i "event persister|task run recorder|scheduler|late runs|cancellation cleanup|repossessor|perpetual|acquired lease|flushing|persisted"

B) Optional: Redis-side visibility (DB 0 for messaging) - Show consumer groups for events stream

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO GROUPS events
docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO CONSUMERS events task-run-recorder
docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO CONSUMERS events event-persister

- Tail the event payloads (non-destructive)

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ \
  | awk '"'"'/^[0-9-]+$/{next} $0=="data"{getline v; print v}'"'"'
'

- Very noisy (use sparingly): see XADD/XREADGROUP/XAUTOCLAIM in real time

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 MONITOR | egrep "XADD|XREADGROUP|XAUTOCLAIM|XACK|XTRIM"

C) Tips - Make sure you’re logging from the pods that actually run the services (API/background services). Workers do not run these server services. - If you split API vs “services-only” deployments, enable the loggers on the services deployment. - Verify services are enabled:

prefect server services ls

- For “scheduled” specifically, the scheduler, recent-deployments scheduler, late-runs, cancellation cleanup, and repossessor will log via Docket/Perpetual. The “docket” logger will show lease acquisition/failover details. If you paste a few lines of your current logs, I can help you tune the filters and confirm you’re seeing the expected messages for each service.

XINFO CONSUMERS only shows metadata about the consumer group (name, idle, pending). It will never show the event payloads. To see the actual event data that Task Run Recorder and Event Persister consume, read from the Redis Stream itself (key: events) with XRANGE/XREAD and filter the data field. Print the last N events’ data payloads

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XRANGE events - + COUNT 10 |
  awk '"'"'
    /^[0-9-]+$/           {next}           # skip message ids
    $0=="data"            {getline v; print v}  # print only the data field value (JSON)
  '"'"'
'

Tail live events (non-destructive “follow”)

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ |
  awk '"'"'
    /^[0-9-]+$/           {next}
    $0=="data"            {getline v; print v}
  '"'"'
'

Filter to only Task Run Recorder–relevant events (prefect.task-run.*) - Add jq to the pipelines above:

... | jq -r 'select(.event | startswith("prefect.task-run"))'

Filter to “scheduled” events (e.g., prefect.flow-run.scheduled)

... | jq -r 'select(.event == "prefect.flow-run.scheduled")'

If your redis-cli supports JSON output (Redis 7+), you can extract fields more cleanly:

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --json XRANGE events - + COUNT 10 |
  jq -r ".[] | {id: .[0], fields: (reduce range(0; (.[1]|length); 2) as \$i ({}; . + { (.[1][\$i]): (.[1][\$i+1]) }))} | .fields.data"
'

Notes - Use DB 0 for messaging (your Docket is on DB 1). - XREAD without GROUP is safe for observing; it won’t claim/ack messages from your services. - XINFO CONSUMERS is still useful to confirm group health: - Task Run Recorder group: task-run-recorder - Event Persister group: event-persister

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO CONSUMERS events task-run-recorder
  docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 XINFO CONSUMERS events event-persister

- If you need to see Redis commands (e.g., XADD/XREADGROUP) briefly:

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 MONITOR | egrep "XADD|XREADGROUP|XAUTOCLAIM|XACK"

Caution: MONITOR is very noisy; use sparingly. If you paste one XRANGE entry (two lines for “data” and its JSON), I can help craft a jq to print exactly the fields you care about (event type, flow/task run ids, etc.).

All three services read the same Redis Stream: events. Use XRANGE/XREAD to print the JSON payloads, then filter by event type for each service. First, confirm the field name holding the JSON (usually data):

docker exec -it <redis-container> redis-cli -a <PASSWORD> -n 0 --raw XRANGE events - + COUNT 1

If you see a field called data with a JSON value, use the commands below. Tail Task Run Recorder events (prefect.task-run.*) - With jq (preferred):

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ \
  | awk '"'"'/^[0-9-]+$/ {next} $0=="data" {getline v; print v}'"'"' \
  | jq -r '"'"'select(.event | startswith("prefect.task-run"))'"'"'
'

- Without jq:

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ \
  | awk '"'"'/^[0-9-]+$/ {next} $0=="data" {getline v; print v}'"'"' \
  | grep '"'"'"event":"prefect.task-run"'"'"'
'

Tail Scheduler “scheduled” events (prefect.flow-run.scheduled) - With jq:

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ \
  | awk '"'"'/^[0-9-]+$/ {next} $0=="data" {getline v; print v}'"'"' \
  | jq -r '"'"'select(.event == "prefect.flow-run.scheduled")'"'"'
'

- Without jq:

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ \
  | awk '"'"'/^[0-9-]+$/ {next} $0=="data" {getline v; print v}'"'"' \
  | grep '"'"'"event":"prefect.flow-run.scheduled"'"'"'
'

Tail all events (what Event Persister will persist) - Raw payloads:

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ \
  | awk '"'"'/^[0-9-]+$/ {next} $0=="data" {getline v; print v}'"'"'
'

- Or pretty-print with jq:

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XREAD BLOCK 0 STREAMS events $ \
  | awk '"'"'/^[0-9-]+$/ {next} $0=="data" {getline v; print v}'"'"' \
  | jq -C .
'

Grab recent entries instead of “tail -f” - Last 50 Task Run Recorder events:

docker exec -it <redis-container> sh -lc '
  redis-cli -a "<PASSWORD>" -n 0 --raw XRANGE events - + COUNT 200 \
  | awk '"'"'/^[0-9-]+$/ {next} $0=="data" {getline v; print v}'"'"' \
  | jq -r '"'"'select(.event | startswith("prefect.task-run")) | .'"'"'
'

Notes - Use DB 0 (messaging). Your Docket is on DB 1. - If your stream stores the JSON under a different field (e.g., payload), replace data in the awk with that field name. - XREAD (without GROUP) is non-destructive; it won’t interfere with your consumer groups.

Short answer: No — putting Docket on Redis DB 1 and running background services in two Kubernetes clusters does not, by itself, cause deadlocks. That setup is correct and recommended. The deadlocks you’re seeing are almost certainly at the Postgres level from concurrent writes by multi-instance consumer services (e.g., Task Run Recorder, Event Persister), not from Docket. How Prefect handles this - Docket (Redis DB 1) coordinates “singleton” services (e.g., Scheduler, Late Runs, etc.) so only one executor runs a keyed perpetual task at a time across all pods/clusters. This prevents duplicate scheduling, but it does not touch your Postgres transactions. - Event-driven “consumer” services (Task Run Recorder, Event Persister, Reactive Triggers) read from the same Redis Stream (events on Redis DB 0) using a shared consumer group across all pods. If a handler raises (e.g., DB deadlock), the message is not acknowledged and will be retried by the consumer framework (up to a max retry count, then DLQ). This provides eventual processing, but you can still see deadlocks and lag until tuning is applied. What actually causes the deadlocks - Concurrent upserts/insertions into the same task_runs/task_run_states rows from multiple recorder instances flushing batches at the same time. - Large DB batches and synchronized flushes increase the overlap window and the chance of lock cycles. What to do - Keep Docket as-is (Redis DB 1, same URL and name across clusters). That is not the problem. - Tune the Task Run Recorder to reduce Postgres lock overlap: - Keep broker reads high, shrink DB write batch:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=100
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=20
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1

- Stagger flush intervals across clusters slightly (e.g., 1.00s vs 1.15s) to avoid synchronized commits. - As a baseline test, run a single global Task Run Recorder (disable in one cluster) to confirm deadlocks stop; then re-introduce the second recorder with smaller DB batch_size. - DB observability: - Enable diagnostics to capture a deadlock graph:

ALTER SYSTEM SET deadlock_timeout = '2s';
    ALTER SYSTEM SET log_lock_waits = on;
    SELECT pg_reload_conf();

- Share one deadlock report; it will name the tables/queries involved and confirm it’s the recorder/persister path. - Infra checks: - Ensure all recorder/persister pods point to the Postgres writer (not a reader or mixed proxy). - If using PgBouncer, prefer session pooling for these services and raise pool size to avoid commit stalls. Bottom line - Docket on DB 1 across clusters is fine and won’t cause Postgres deadlocks. - Deadlocks arise from concurrent DB writes by multi-instance consumer services; Prefect will retry unacked messages, but you should tune batch sizes/flush timing (or temporarily reduce to one recorder) to eliminate the contention.

Short answer: seeing a “Completed” event before the DB shows the completed state can happen briefly because events and DB writes are not atomic. But if it lasts more than a few seconds, it’s usually backlog/retry in Task Run Recorder (client-orchestrated path) or Postgres deadlocks. Here’s how to tell which path you’re on and how to fix it. First, determine how the state is supposed to reach Postgres - Server-orchestrated tasks: the engine calls the API

…/task_runs/<id>/set_state

and the API writes the state synchronously. Events are published in parallel, so you can momentarily see the event before the DB commit lands — but the DB should update within seconds unless you’re reading from a replica or the API isn’t getting the set_state calls. - Client-orchestrated tasks: Task Run Recorder consumes

prefect.task-run.*

events and writes task_runs/task_run_states in bulk. In this path, “Background Services received event” just means it was delivered; it’s still subject to batching, deadlocks, and retries before the DB reflects the new state. Quick triage: which path is yours? - Check API logs around the time of completion for set_state: - Look for

POST /api/task_runs/<id>/set_state

and a 2xx response. If present, you’re server-orchestrated and the DB should reflect immediately. If missing, you’re relying on Task Run Recorder. - If you’re using client orchestration or can’t confirm set_state calls, proceed with the recorder checks below. If using Task Run Recorder, verify and tune - Check if the event was delivered but is still pending (not acked):

# On the messaging DB (0)
  redis-cli -u "<redis://host:6379/0>" XPENDING events task-run-recorder
  redis-cli -u "<redis://host:6379/0>" XINFO CONSUMERS events task-run-recorder

High pending counts or long idle times mean the recorder hasn’t flushed/acked yet. - Look for recorder flush/retry logs: - Enable:

PREFECT_LOG_LEVEL=DEBUG
    PREFECT_LOGGING_EXTRA_LOGGERS="prefect.server.services.task_run_recorder,prefect.server.events.messaging,prefect_redis.messaging"

- You should see “read N messages”, “flushed N task runs / M states”, or retries on errors (e.g., DeadlockDetectedError). - Reduce Postgres lock overlap (common cause for delays with multiple recorder pods): - Keep broker reads high, write smaller DB batches, and stagger flush intervals across clusters:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=100
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=20
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1      # cluster A
    # cluster B (stagger)
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1.15

- Baseline test: temporarily run a single global Task Run Recorder (disable in one cluster). If the gap disappears, the issue is cross-pod overlap/deadlocks; keep batch_size modest (10–20) and add the second recorder back. Database-side checks - Verify the state row exists in task_run_states and whether task_runs has the pointer updated:

-- Replace with your task run id
  SELECT id, state_type, name, timestamp
  FROM task_run_states
  WHERE task_run_id = '<TASK_RUN_ID>'
  ORDER BY timestamp DESC
  LIMIT 5;

  SELECT state_id, state_type, state_timestamp
  FROM task_runs
  WHERE id = '<TASK_RUN_ID>';

If there’s no new state row, the recorder hasn’t written it yet (backlog/retry). If a Completed state exists but task_runs didn’t advance, share the deadlock logs; in practice, both operations occur in the same transaction, so you’ll usually see neither when a deadlock rolls back. - Turn on Postgres deadlock diagnostics to capture one cycle:

ALTER SYSTEM SET deadlock_timeout = '2s';
  ALTER SYSTEM SET log_lock_waits = on;
  SELECT pg_reload_conf();

Then share one deadlock report so we can confirm it’s the recorder upsert and tune batch sizes further. Also rule out these environment pitfalls

- Read-after-write lag: ensure every API/recorder/persister pod points to the writer endpoint for Postgres (not a reader or mixed proxy). - Redis groups: confirm a single consumer group for the recorder on the events stream (you already have a single broker, good):

redis-cli -u "<redis://host:6379/0>" XINFO GROUPS events

- API vs services segregation: if you split “services-only” pods from API pods, make sure recorder is enabled and logging on the services pod and not accidentally disabled. Bottom line - A tiny window where an event is visible before the DB state commits is normal. Longer gaps indicate the recorder hasn’t flushed yet or is retrying after a deadlock. - Keep recorder read_batch_size high, DB batch_size modest (10–20), stagger flush intervals across clusters, and confirm all writers hit the same Postgres writer. Enable DEBUG logs and Postgres deadlock diagnostics to verify the path. If you can share a deadlock snippet or a few lines of recorder DEBUG logs around the event, I’ll suggest precise next-step values.

Thanks for the detail — a consistent ~15 minute gap strongly suggests the Task Run Recorder is not committing promptly due to retries/backoff after write contention (e.g., deadlocks) or because messages are sitting “pending” in the consumer group before being reprocessed. Here’s a focused runbook to confirm the cause and fix it. Step 1 — Verify if messages are stuck pending in Redis - On your messaging DB (0), check pending for the recorder group:

redis-cli -u "<redis://host:6379/0>" XPENDING events task-run-recorder
redis-cli -u "<redis://host:6379/0>" XPENDING events task-run-recorder - + 20
redis-cli -u "<redis://host:6379/0>" XINFO CONSUMERS events task-run-recorder

- What to look for: - Large pending count and very high idle times (approaching the 15m window) = messages were claimed but not acked (recorder isn’t flushing). - If you see one consumer with huge pending and long idle, that pod may be failing/retrying and holding the batch. Step 2 — Confirm recorder is retrying/blocked (enable DEBUG logs) - On your API/background-service pods:

PREFECT_LOG_LEVEL=DEBUG
PREFECT_LOGGING_EXTRA_LOGGERS="prefect.server.services.task_run_recorder,prefect.server.events.messaging,prefect_redis.messaging"

- Restart pods and watch for: - “read N messages”, “flushed N task runs / M states” - Deadlock/retry messages (e.g., DeadlockDetectedError) - Backoff logs between retries Step 3 — Apply targeted tuning (reduce overlap, keep throughput) - Keep broker reads high (drain Redis quickly), write smaller DB batches (shorter transactions), and stagger flushes across clusters: - Cluster A

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=100
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=20
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1.00

- Cluster B

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=100
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=20
    PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1.15

- This reduces row lock overlap across recorder pods and avoids synchronized commits. Step 4 — Ensure the consumer framework will reprocess “stuck” messages quickly - Confirm these (defaults are typically fine, but set explicitly to be sure):

PREFECT_REDIS_MESSAGING_CONSUMER_SHOULD_PROCESS_PENDING_MESSAGES=true
PREFECT_REDIS_MESSAGING_CONSUMER_MIN_IDLE_TIME=5000

- This ensures pending messages are revisited quickly if a consumer idles/crashes. Step 5 — Database visibility and safety guards - Enable deadlock diagnostics so we can see the exact cycle and confirm it’s the recorder upserts:

ALTER SYSTEM SET deadlock_timeout = '2s';
ALTER SYSTEM SET log_lock_waits = on;
SELECT pg_reload_conf();

- Check the state rows when the gap occurs:

SELECT id, state_type, name, timestamp
FROM task_run_states
WHERE task_run_id = '<TASK_RUN_ID>'
ORDER BY timestamp DESC
LIMIT 5;

SELECT state_id, state_type, state_timestamp
FROM task_runs
WHERE id = '<TASK_RUN_ID>';

- If neither table has the new state during the delay, the recorder hasn’t committed yet (retry/backoff). If you’re seeing frequent deadlocks, consider further reducing

batch_size

(e.g., 10–15) and/or temporarily running a single global recorder to validate. Optional — One-recorder baseline - Temporarily disable the recorder in one cluster and keep a single recorder running. If the 15m gap disappears, the cause is cross-pod write overlap. Reintroduce the second recorder with small

batch_size

and staggered

flush_interval

. Other sanity checks - All recorder pods must point to the Postgres writer (not a reader or mixed proxy). - If PgBouncer is in the path, prefer session pooling for recorder connections and ensure pool size is sufficient. If you can share: - Output of XPENDING/XINFO for the recorder group during the gap, - A few DEBUG log lines from the recorder around a delayed commit, - One Postgres deadlock report,

…I’ll give you precise next-step values (batch sizes, replica count, or additional DB tweaks) to eliminate the 15-minute lag.

Thanks — what you’re seeing is a classic side effect of out‑of‑order event delivery plus backlog: a “RUNNING” event can be ingested before the “PENDING” event for the same task run. The Task Run Recorder uses “latest state wins by timestamp,” so: - If RUNNING arrives first, it will upsert RUNNING. - When PENDING arrives later, it’s older, so it’s ignored. - When COMPLETED finally arrives (newest timestamp), the recorder upserts COMPLETED and the state corrects. Why it happens - Events are published by different processes (API, worker) and batched; there is no global ordering guarantee across processes. - Network/jitter and Redis batching/flush intervals can reorder delivery. - Clock skew across nodes can make a RUNNING event’s timestamp appear newer than PENDING even if created earlier. - Under load, the recorder may be retrying (e.g., deadlocks), so COMPLETED lands later, leaving RUNNING visible until the retry succeeds. What to do to reduce the visible gap - Ensure tight time sync (important) - NTP on all API/worker/service nodes. Even small skew can make timestamps arrive “out of order.” - Keep recorder flushes fast and staggered across clusters - High read, modest DB batch, small, slightly different intervals across clusters: - Cluster A:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=100
      PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=20
      PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1.00

- Cluster B:

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=100
      PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=20
      PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1.15

- This reduces DB lock overlap and avoids synchronized commits. - Keep the publisher’s flush cadence short (less batching, less reordering) - On API/worker processes that publish events:

PREFECT_REDIS_MESSAGING_PUBLISHER_BATCH_SIZE=50
    PREFECT_REDIS_MESSAGING_PUBLISHER_PUBLISH_EVERY=1

- Confirm the recorder is healthy (to avoid 15m “corrections”) - Redis DB 0: -

XINFO GROUPS events

and

XINFO CONSUMERS events task-run-recorder

-

XPENDING events task-run-recorder

(pending size/idle should not balloon) - Enable DEBUG logs for the recorder to see “read N messages” / “flushed N task runs” and any retries:

PREFECT_LOG_LEVEL=DEBUG
    PREFECT_LOGGING_EXTRA_LOGGERS="prefect.server.services.task_run_recorder,prefect.server.events.messaging,prefect_redis.messaging"

- Database deadlock/backoff visibility - If you still see multi‑minute “RUNNING until it corrects,” the COMPLETED upsert is likely retrying after deadlocks. Enable Postgres diagnostics:

ALTER SYSTEM SET deadlock_timeout = '2s';
    ALTER SYSTEM SET log_lock_waits = on;
    SELECT pg_reload_conf();

- If deadlocks appear, keep DB batch_size = 10–20, and consider temporarily running a single global recorder to validate the fix, then add the second back. If you want to avoid reliance on event order entirely - Ensure your runs use server‑side state setting (

…/task_runs/<id>/set_state

); those DB writes are synchronous and don’t depend on event ordering. The Task Run Recorder is only authoritative for client‑orchestrated task events. If your logs show no

set_state

calls, you’re on the client path and the recorder is the authority. Sanity checks - All recorder pods point to the same Postgres writer (not a reader/mixed proxy). - Single Redis broker and single consumer group for the recorder (task-run-recorder). - No big clock skew across clusters. If you can share: - A brief recorder DEBUG log window around one run (showing read/flush/retry), - XPENDING summary during the gap, - One Postgres deadlock report (if any), …I can confirm whether backlog/retry is the culprit and suggest exact batch sizes/replica counts to eliminate the “RUNNING until it corrects” window.