Thanks for flagging this. In Prefect 3.x HA, task/flow state changes emit events that are published to a “broker” (Redis Streams when using prefect-redis), then consumed by background services (e.g., event persister, task-run recorder). If Redis doesn’t receive an event, downstream services won’t react and task runs can appear stuck in RUNNING. Quick architecture recap: - Publisher writes events to Redis stream named

events

- Consumer groups read that stream: -

event-persister

(writes events to DB) -

task-run-recorder

(reconciles task run state from events) -

reactive-triggers

,

event-logger

(optional/debug) Given your symptom (“API server published task run event, but Redis didn’t receive it”), the most common causes are: - The API isn’t actually using the Redis broker (default is in-memory per-process) - Redis connectivity/credentials/SSL mismatch on the API container(s) - The prefect-redis extra isn’t installed in the API image - Background services not enabled/running (especially event persister and task-run recorder) - Redis Streams consumer group issues (stuck pending, DLQ growth) Targeted checks 1) Confirm you’re using the Redis broker on the API pod(s) - This should show broker settings and their sources:

prefect config view --show-defaults --show-sources

Look for: -

PREFECT_SERVER_EVENTS__MESSAGING_BROKER = prefect_redis.messaging

-

PREFECT_SERVER_EVENTS__MESSAGING_CACHE = prefect_redis.messaging

If these are not set, you’re on the in-memory broker and HA will be inconsistent. 2) Confirm the prefect-redis extra is installed on API containers - Your API image or Helm values need to include

prefect[redis]

- If it’s missing, the broker setting can silently fall back or error in logs 3) Verify Redis connectivity from API pods - Network/SSL/creds must match your Redis deployment. If you have shell access:

redis-cli -u redis://<user>:<pass>@<host>:<port>/<db> PING

- If you use TLS, ensure the URL is rediss:// and SSL=true (depending on your config) 4) Inspect the Redis streams directly - Stream length and recent entries:

redis-cli XLEN events
redis-cli XRANGE events - + COUNT 5

- Consumer groups and consumers:

redis-cli XINFO GROUPS events
redis-cli XINFO CONSUMERS events event-persister
redis-cli XINFO CONSUMERS events task-run-recorder

If XLEN stays 0 while you’re generating state changes, the publisher is not reaching Redis. 5) Confirm background services are enabled and healthy - Event persister and task-run recorder must be on: -

PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED=true

-

PREFECT_API_SERVICES_TASK_RUN_RECORDER_ENABLED=true

- Check API logs for lines like “Event persister started”, “Task run recorder started”, and any Redis connection errors/timeouts. 6) Optional broker tuning for reliability under load - Reduce batching delay to flush more frequently: -

PREFECT_REDIS_MESSAGING_PUBLISHER_PUBLISH_EVERY=1

- Increase event persister queue size if you see queue pressure warnings: -

PREFECT_API_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=50000

(or higher per your needs) What I’d like from you to help pinpoint the issue - How is Prefect deployed? (Helm chart + version/values or custom manifests) - The relevant environment variables from your API pods (redact secrets), especially: -

PREFECT_SERVER_EVENTS__MESSAGING_BROKER

-

PREFECT_SERVER_EVENTS__MESSAGING_CACHE

- Any

PREFECT_REDIS_*

vars you’re using (URL/host/port/ssl) -

PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED

-

PREFECT_API_SERVICES_TASK_RUN_RECORDER_ENABLED

- Snippets from API logs around the time of the stuck run that include “event”, “redis”, “persister”, or “task run recorder” - Output of: -

redis-cli XLEN events

-

redis-cli XINFO GROUPS events

-

redis-cli XINFO CONSUMERS events event-persister

-

redis-cli XINFO CONSUMERS events task-run-recorder

Notes and related context

- This open report describes a related symptom where tasks remained RUNNING when flows crashed via K8s OOM; it may help cross-check your behavior: Tasks stuck in Running state while flow is in Crashed state - If you’re on 3.6.24, consider upgrading to the latest 3.6.x; we’ve made ongoing reliability improvements to event propagation and observers in patch releases. Latest notes: 3.6.26 changelog If you can share the details above, I can help zero in on whether the publisher isn’t using Redis, the connection is failing, or a consumer group is stuck—and suggest the exact remediation.

Got it — if the API is definitely configured to use the Redis broker, intermittent “published but never shows up in Redis” almost always comes down to one of these in practice: - In-flight publisher buffer lost during API pod restart/rolling updates (batching + periodic flush) - Brief Redis connectivity blips during/after DB state commit (publish is fire-and-forget) - Publisher-side deduplication filtering more than intended - A Redis/TLS misconfig that only fails intermittently under load (and logs the error, but the event is gone) Why this can happen - Prefect emits events from an “after_transition” hook and hands them to an async publisher that batches and flushes on an interval. Default publisher settings typically batch multiple events and flush every ~10s. If the process exits or restarts before the next flush, any in‑memory events can be lost. If there’s a short network blip right as we attempt XADD, that publish can fail and won’t be retried synchronously. - If

DEDUPLICATE_BY

was set for the publisher, events can be dropped as “duplicates” unintentionally. - None of the above affects the DB state that was already committed — so a run can remain “RUNNING” if the downstream consumers never see the transition event. Concrete mitigations to minimize/stop drops 1) Flush more aggressively to Redis - Make the publisher nearly synchronous by shrinking the batch/flush window:

PREFECT_REDIS_MESSAGING_PUBLISHER_BATCH_SIZE=1
PREFECT_REDIS_MESSAGING_PUBLISHER_PUBLISH_EVERY=1

This reduces the time window where an API restart or brief network issue can drop events. 2) Disable any accidental dedup

PREFECT_REDIS_MESSAGING_PUBLISHER_DEDUPLICATE_BY=

Leave it unset/empty unless you explicitly want deduplication. 3) Ensure graceful shutdown for API pods (Kubernetes) - Increase

terminationGracePeriodSeconds

(e.g., 30–60s) and add a small

preStop

hook (e.g.,

sleep 5

) to allow the publisher to flush on shutdown. - Avoid simultaneous restarts of all API replicas; roll slowly. 4) Turn on DEBUG and watch for publish failures - On API pods:

PREFECT_LOG_LEVEL=DEBUG

- Look for lines containing “Event”, “publish”, “redis”, or tracebacks around XADD errors. If you see intermittent write errors/timeouts, that confirms a transient publish failure. 5) Double-check Redis/TLS and keepalives - If using TLS, ensure the client is using the right URI (rediss://) and SSL=true in the redis settings. Keep the

health_check_interval

low enough to keep the connection alive (default is fine, but you can tune if your infra is finicky). - Sanity check with a lightweight publish test during your normal load window (see quick test below). 6) Make sure the downstream “safety net” is enabled - Task run recorder helps reconcile stragglers and “lost follower” events:

PREFECT_API_SERVICES_TASK_RUN_RECORDER_ENABLED=true

It won’t fix a completely missing publish, but it reduces lingering RUNNING states after crashes. Quick verification tests - Emit a test event and watch Redis:

prefect events emit --event test.ping --resource "prefect.resource:api"
redis-cli XLEN events
redis-cli XRANGE events - + COUNT 5

- Generate a few task state transitions and immediately roll a single API pod to see if any publishes are lost. If tightening the publisher settings fixes it, you’ve found the culprit. If you’re on 3.6.24, I also recommend bumping to the latest 3.6.x — we’ve made ongoing reliability improvements around event/observer propagation. Changelog: 3.6.26 If you can share: - Your current values for: -

PREFECT_REDIS_MESSAGING_PUBLISHER_BATCH_SIZE

-

PREFECT_REDIS_MESSAGING_PUBLISHER_PUBLISH_EVERY

-

PREFECT_REDIS_MESSAGING_PUBLISHER_DEDUPLICATE_BY

- Whether API pods are rolling/restarting around the times you see missing events - Any DEBUG log snippets showing Redis/publish errors …I can help you tune these precisely for your environment.