sometimes people have intermittent cnx issues with websockets - how many tasks are we talking? are any

task-run.Completed

events coming through?

The flow I'm testing this with has 12 tasks, I wasn't able to see any of the task-run.Completed events coming through, but the UI does update after about 15 minutes to show they completed

I wasn't able to see any of the task-run.Completed events coming through, but the UI does update after about 15 minutes to show they completed

hm, it sounds like they eventually showed up then?

@Marvin I tried this on prefect 3.4.10 for the server, workers, and flow run image, as well as prefect 3.4.13. It was working fine on both with the basic helm install, but the issue arose when I converted to high availability setup with 3 api servers, separate background services, a redis cluster in kubernetes, and a separate postgres cluster with patroni outside of kubernetes. For redis I tried using the Redis Enterprise Cluster setup as well as the bitnami Redis master/slave setup. Both seemed to function aside from this issue. I am using an nginx ingress for accessing the UI, but all api server connections should be using internal kubernetes service addresses without a 3rd party ingress.

Thanks — that narrows it down. Given it worked with the basic Helm install and only broke after you switched to HA with Redis + split background services, this is almost certainly your events pipeline in HA not behaving — i.e., workers’ event acks aren’t being processed, so tasks stay Pending until the server’s reconciliation service eventually catches up. Key HA-specific culprits to check - Redis messaging config consistency - Ensure ALL server components (every API pod and every background service pod) have the same messaging settings and point to the same Redis instance/db: -

PREFECT_MESSAGING_BROKER=prefect_redis.messaging

-

PREFECT_MESSAGING_CACHE=prefect_redis.messaging

-

PREFECT_REDIS_MESSAGING_HOST

,

PREFECT_REDIS_MESSAGING_PORT

,

PREFECT_REDIS_MESSAGING_DB

, and auth/SSL vars as needed - Verify the prefect-redis integration is installed in the images running API and background services. If it isn’t, Prefect can silently fall back to in-memory messaging on some pods, which breaks acks in HA. - Redis mode support - Prefect’s Redis messaging uses Redis Streams semantics. Redis “cluster” mode can be… uncooperative with streams unless keys are carefully hash-tagged to the same slot. If you’re on Redis Enterprise Cluster or OSS cluster, try: - Pointing Prefect at a proxy/single endpoint that presents a non-sharded view, or - Testing with a single-writer Redis (Bitnami primary or Enterprise with a proxy endpoint) to confirm cluster sharding is the cause. - There’s an open report where Redis messaging caused tasks/subflows to go missing in the UI: Issue #18540. Your symptoms are very similar. - TaskRunRecorder running and connected - In HA, the TaskRunRecorder service must be running in your background services and reading from the same messaging broker as the API pods. If it’s not running or cannot connect to Redis, task events won’t be recorded promptly (hence Pending in UI). - Check logs in the background services for: - “TaskRunRecorder started” - Connection errors to Redis - Backlogs/consumer errors - Prefect server services overview: Server (Helm) docs - API WebSocket path and acks - Even though you’re not using ingress for API, confirm worker logs show “Websocket connected” with stable connections to the API Service and not constant reconnects. - Since you have multiple API pods, make sure all API pods are identically configured for messaging. If some API pods run with in-memory messaging while others use Redis, acks will be inconsistent. Quick isolations to prove the cause - Temporarily disable Redis messaging (let Prefect default to in-memory) across all pods in HA. If the issue disappears, Redis messaging config or Redis mode is the culprit. - Keep Redis messaging on but point to a single non-cluster Redis (Bitnami primary only) and retest. If the problem vanishes, it’s Redis Cluster behavior with streams. Also, stay current - You tested 3.4.10 and 3.4.13. Keep 3.4.13+ for the base-path/internal client fixes and other services reliability improvements: - Latest releases: Release notes If you can share: - The exact env vars on an API pod and on a background service pod (the

PREFECT_MESSAGING_*

and

PREFECT_REDIS_*

set) - Whether your Redis endpoint is true Cluster mode vs a proxy/single endpoint - Logs from the TaskRunRecorder pod during a run …I can call out the exact misconfiguration. But right now, my money’s on mixed messaging backends across pods or Redis Cluster stream behavior. Delightful, I know.

Got it — thanks for the details. The giveaway is your background-services pod not logging “TaskRunRecorder started” (or anything about task_run_recorder) and only showing Redis stream housekeeping. In HA, that means your recorder service isn’t actually running, so worker-emitted task events never get recorded — hence tasks linger in Pending until the eventual reconciliation pass. What to check/fix - Force-enable the recorder (and friends) - Add these env vars to the background-services deployment and restart: -

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

-

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=true

-

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true

- After restart, you should see logs like “TaskRunRecorder started” in that pod. If not, it’s not being launched. - Ensure the background-services pod has the same DB config as API pods - It must write to the same Postgres as the API. If the DB URL isn’t set there, the recorder can’t persist runs. - Confirm your Postgres env (e.g.,

PREFECT_API_DATABASE_CONNECTION_URL

) is present in the background-services pod. - Redis messaging config consistency - You’ve set broker/cache to Redis — good. Explicitly add

PREFECT_REDIS_MESSAGING_DB

(e.g.,

0

) so every pod uses the same DB. - The log “No active consumer groups found for stream actions” suggests no consumer is running for that stream — i.e., your background services that should consume from Redis aren’t actually up (again pointing at services not starting). - Make sure the background-services image includes the redis integration - The log line from

prefect.prefect_redis.messaging

implies it’s there, but double-check the image build. Official images need the

prefect-redis

extra or package installed (e.g.,

pip install 'prefect-redis>=0.2'

or use

prefect[redis]

) for all server images, not just workers. - Sanity check WebSocket connectivity from workers - In worker logs you should see “Websocket connected” and not a constant reconnect loop. Unconfirmed events usually mean the server isn’t acking — because the recorder/event services aren’t alive. Useful references - Settings reference (service enable flags, DB, messaging): Settings reference - Helm/server overview: Server (Helm) docs - Related Redis messaging issue (symptoms match): Issue #18540 If you share: - The background-services Deployment spec (image, command/args, env) - Whether

PREFECT_API_DATABASE_CONNECTION_URL

is set in that pod - A few minutes of its logs after restart …I can pinpoint why the recorder isn’t starting. But right now, the fix is to explicitly enable TaskRunRecorder and ensure that pod can reach both Redis and your Postgres. Without the recorder running, the UI will continue to contemplate “Pending” for far too long, much like the rest of us.

Short answer: use non-clustered Redis with persistence and a single stable endpoint. Prefect’s events/messaging pipeline uses Redis Streams; cluster mode tends to cause grief (hash slots, multi-stream ops) and leads to exactly the symptoms you saw. Recommended HA patterns for Prefect 3 - Topology - Preferred: a non-clustered Redis with persistence (AOF) and HA failover - Options: - Managed: AWS ElastiCache (cluster mode disabled), GCP Memorystore (Standard), Azure Cache for Redis (non-clustered). - Self-hosted: Bitnami Redis with Sentinel (primary + replicas + automatic failover). - Avoid: Redis Cluster (cluster mode enabled). Prefect does not guarantee stream keys land in the same slot; multi-stream consumer behavior may trigger MOVED errors or partial consumption. If you must use Enterprise/Cluster, use a proxy/single endpoint that abstracts sharding. - Redis configuration - Persistence: enable AOF (appendonly yes) so consumer groups and offsets survive restarts. - Memory policy: use noeviction (or at least do not evict stream keys). Evicting streams will cause dropped events and Pending tasks. - Keepalive/timeouts: sane defaults (e.g., tcp-keepalive 300). - Security/TLS: as required for your environment. - Prefect configuration (all API pods + background-services pods) - Ensure the redis integration is installed in those images (e.g., prefect-redis or prefect[redis]). - Set the same messaging environment in every server-facing pod: -

PREFECT_MESSAGING_BROKER=prefect_redis.messaging

-

PREFECT_MESSAGING_CACHE=prefect_redis.messaging

-

PREFECT_REDIS_MESSAGING_HOST=redis.svc.cluster.local

-

PREFECT_REDIS_MESSAGING_PORT=6379

-

PREFECT_REDIS_MESSAGING_DB=0

-

PREFECT_REDIS_MESSAGING_USERNAME=default

(if applicable) -

PREFECT_REDIS_MESSAGING_PASSWORD=...

- Make sure background-services is actually running the recorder: -

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

- Also keep scheduler/repossessor enabled in HA: -

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=true

-

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true

- And that background-services has the same Postgres URL as API pods. - What “healthy” looks like - Background-services logs should show the recorder starting (not just Redis group housekeeping). - Worker logs should show “Websocket connected” and minimal “unconfirmed events”. - Redis should show consumer groups on expected streams: -

XINFO GROUPS events

-

XINFO GROUPS actions

-

XPENDING events

should not trend upward indefinitely. - Kubernetes notes - Service endpoints only; you don’t need ingress for API-to-Redis. - If you use Sentinel, point Prefect at the sentinel-aware endpoint (managed by your chart) so failover is automatic but the hostname remains stable. - Keep Redis in the same cluster/region/namespace to minimize latency. If you want, I can share a minimal Bitnami Redis + Sentinel values sketch or check your background-services Deployment to confirm the recorder and messaging settings are wired correctly. References: - Settings reference: Settings reference - Server/Helm overview: Server (Helm) docs - Related Redis messaging issue: Issue #18540 I realize “disable Redis and it works” is tempting. For HA, though, a single endpoint, non-clustered Redis with persistence and the recorder service actually running is the boring, reliable option—my favorite kind.

you will have consistency issues if you run many servers without a distributed message broker

@Marvin I set up a single redis master node in my prefect namespace for testing, but it still isnt working with these settings: env: - name: PREFECT_API_DATABASE_MIGRATE_ON_START value: "false" - name: PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED value: "true" - name: PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED value: "true" - name: PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED value: "true" - name: PREFECT_MESSAGING_BROKER value: "prefect_redis.messaging" - name: PREFECT_MESSAGING_CACHE value: "prefect_redis.messaging" - name: PREFECT_REDIS_MESSAGING_HOST value: "redis-master.prefect.svc.cluster.local" - name: PREFECT_REDIS_MESSAGING_PORT value: "6379" - name: PREFECT_REDIS_MESSAGING_PASSWORD value: "..."

Thanks — that helps. Given you’ve enabled the services and switched to a single Redis master, but it still behaves the same, we need to prove two things: 1) The API pods can publish events to Redis 2) The background-services pod is actually consuming them (TaskRunRecorder running) Right now your logs suggest Redis is reachable, but nothing is consuming “events” (and workers keep piling up “unconfirmed events” because the API isn’t acking/persisting). Do these quick checks: - Verify effective settings inside both an API pod and the background-services pod - Run this in each pod:

prefect config view --show-defaults --show-sources

- Confirm these are present and identical in both: -

PREFECT_MESSAGING_BROKER=prefect_redis.messaging

-

PREFECT_MESSAGING_CACHE=prefect_redis.messaging

-

PREFECT_REDIS_MESSAGING_HOST=redis-master.prefect.svc.cluster.local

-

PREFECT_REDIS_MESSAGING_PORT=6379

-

PREFECT_REDIS_MESSAGING_DB=0

← you’re missing this, set it explicitly -

PREFECT_REDIS_MESSAGING_USERNAME

(set to

default

if your Redis uses ACLs) -

PREFECT_REDIS_MESSAGING_PASSWORD=...

- If any API pod is missing these, it’ll publish to in-memory and never reach Redis. - Confirm the recorder is running in background-services - With

DEBUG

logging, you should see something like “TaskRunRecorder started”. - If you don’t, the service isn’t starting. Keep: -

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

-

PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=true

-

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true

- Check Redis for streams and groups - From a shell with redis-cli access:

XINFO STREAM events
    XINFO GROUPS events
    XINFO STREAM actions
    XINFO GROUPS actions

- You should see consumer groups for Prefect. If “events” has no groups while “actions” does, your recorder isn’t attached to events, or the distributor isn’t pushing. - Look for publish/consumer errors in API and background logs - API pods: search for errors from

prefect.server.events.messaging

when creating publishers or publishing (if they can’t publish, they won’t ack workers → “unconfirmed events”). - Background-services: search for

TaskRunRecorder

and any consumer errors. - Ensure prefect-redis is installed in your server images - Inside an API and the background pod:

python -c "import prefect.prefect_redis; print('ok')"

- If that import fails, the env vars will be ignored and you’ll silently fall back to in-memory on that pod. Install the extra in the image (e.g., pip install 'prefect-redis>=0.2' or use a Prefect image built with the redis extra). Minimal env set I recommend on all server-side pods (API + background-services)

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=true
PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true

PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_REDIS_MESSAGING_HOST=redis-master.prefect.svc.cluster.local
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# If Redis ACLs are enabled:
# PREFECT_REDIS_MESSAGING_USERNAME=default
PREFECT_REDIS_MESSAGING_PASSWORD=...

If, after this, you still don’t see “TaskRunRecorder started”, paste the first 100 lines of the background-services logs after a restart. If “events” has no consumer groups, paste

XINFO GROUPS events

. We’ll zero in on whether the API isn’t publishing or the recorder isn’t consuming. Yes, it’s a bit of “everything must match.” HA is thrilling like that.

Your message was too long, here's your message at the allowed limit: I see lots of activity via the redis-cli but still no tasks in the prefect ui 1755714088 778886 [0 192 168 241 233 42780] XAUTOCLAIM actions actions actions-prefect-server-5bbcd44c5b-nns8z-7-a70ec10b 5000 0-0 COUNT 1 1755714088 779271 [0 192 168 225 156 33064] XAUTOCLAIM events event-persister event-persister-prefect-server-5bbcd44c5b-nsf4p-7-dde07edc 5000 0-0 COUNT 1 1755714088 779337 [0 192 168 61 229 47252] XAUTOCLAIM events reactive-triggers reactive-triggers-prefect-server-5bbcd44c5b-fx8lq-7-decb7016 5000 0-0 COUNT 1 1755714088 779461 [0 192 168 241 233 42780] XREADGROUP GROUP actions actions-prefect-server-5bbcd44c5b-nns8z-7-a70ec10b COUNT 1 BLOCK 1000 STREAMS actions > 1755714088 779998 [0 192 168 225 156 33064] XREADGROUP GROUP event-persister event-persister-prefect-server-5bbcd44c5b-nsf4p-7-dde07edc COUNT 1 BLOCK 1000 STREAMS events > 1755714088 780034 [0 192 168 61 229 39250] XAUTOCLAIM events task-run-recorder task-run-recorder-prefect-server-5bbcd44c5b-fx8lq-7-f3e9b917 5000 0-0 COUNT 1 1755714088 780048 [0 192 168 61 229 47252] XREADGROUP GROUP reactive-triggers reactive-triggers-prefect-server-5bbcd44c5b-fx8lq-7-decb7016 COUNT 1 BLOCK 1000 STREAMS events > 1755714088 780565 [0 192 168 61 229 39250] XREADGROUP GROUP task-run-recorder task-run-recorder-prefect-server-5bbcd44c5b-fx8lq-7-f3e9b917 COUNT 1 BLOCK 1000 STREAMS events > 1755714088 987128 [0 192 168 241 234 43272] XAUTOCLAIM events task-run-recorder task-run-recorder-prefect-server-background-services-765b67498b-hjvz7-7-df704d82 5000 0-0 COUNT 1 1755714088 987193 [0 192 168 241 233 42798] XAUTOCLAIM events reactive-triggers reactive-triggers-prefect-server-5bbcd44c5b-nns8z-7-1a456ebe 5000 0-0 COUNT 1 1755714088 987313 [0 192 168 241 234 43252] XAUTOCLAIM events event-persister event-persister-prefect-server-background-services-765b67498b-hjvz7-7-195adb67 5000 0-0 COUNT 1 1755714088 987514 [0 192 168 241 233 42798] XREADGROUP GROUP reactive-triggers reactive-triggers-prefect-server-5bbcd44c5b-nns8z-7-1a456ebe COUNT 1 BLOCK 1000 STREAMS events > 1755714088 987560 [0 192 168 241 234 43272] XREADGROUP GROUP task-run-recorder task-run-recorder-prefect-server-background-services-765b67498b-hjvz7-7-df704d82 COUNT 1 BLOCK 1000 STREAMS events > 1755714088 987685 [0 192 168 225 156 33070] XAUTOCLAIM events task-run-recorder task-run-recorder-prefect-server-5bbcd44c5b-nsf4p-7-61fbe48a 5000 0-0 COUNT 1 1755714088 987701 [0 192 168 61 229 39238] XAUTOCLAIM events event-persister event-persister-prefect-server-5bbcd44c5b-fx8lq-7-5a6f6553 5000 0-0 COUNT 1 1755714088 987785 [0 192 168 225 156 33098] XAUTOCLAIM events reactive-triggers reactive-triggers-prefect-server-5bbcd44c5b-nsf4p-7-df036a9e 5000 0-0 COUNT 1 1755714088 987806 [0 192 168 241 234 43252] XREADGROUP GROUP event-persister event-persister-prefect-server-background-services-765b67498b-hjvz7-7-195adb67 COUNT 1 BLOCK 1000 STREAMS events > 1755714088 988285 [0 192 168 61 229 39238] XREADGROUP GROUP event-persister event-persister-prefect-server-5bbcd44c5b-fx8lq-7-5a6f6553 COUNT 1 BLOCK 1000 STREAMS events > 1755714088 988295 [0 192 168 225 156 33070] XREADGROUP GROUP task-run-recorder task-run-recorder-prefect-server-5bbcd44c5b-nsf4p-7-61fbe48a COUNT 1 BLOCK 1000 STREAMS events > 1755714088 988480 [0 192 168 225 156 33098] XREADGROUP GROUP reactive-triggers reactive-triggers-prefect-server-5bbcd44c5b-nsf4p-7-df036a9e COUNT 1 BLOCK 1000 STREAMS events > 1755714089 088651 [0 192 168 61 229 39220] XAUTOCLAIM actions actions actions-prefect-server-5bbcd44c5b-fx8lq-7-7e4bbc63 5000 0-0 COUNT 1 1755714089 089135 [0 192 168 61 229 39220] XREADGROUP GROUP actions actions-prefect-server-5bbcd44c5b-fx8lq-7-7e4bbc63 COUNT 1 BLOCK 1000 STREAMS actions > 1755714089 189259 [0 192 168 225 156 33054] XAUTOCLAIM actions actions actions-prefect-server-5bbcd44c5b-nsf4p-7-e0a0d368 5000 0-0 COUNT 1 1755714089 189866 [0 192 168 225 15

6 33054] XREADGROUP GROUP actions actions-prefect-server-5bbcd44c5b-nsf4p-7-e0a0d368 COUNT 1 BLOCK 1000 STREAMS actions > 1755714089 289747 [0 192 168 241 234 43268] XAUTOCLAIM actions actions actions-prefect-server-background-services-765b67498b-hjvz7-7-eded7241 5000 0-0 COUNT 1 1755714089 290232 [0 192 168 241 234 43268] XREADGROUP GROUP actions actions-prefect-server-background-services-765b67498b-hjvz7-7-eded7241 COUNT 1 BLOCK 1000 STREAMS actions > 1755714089 423014 [0 127 0 0 1 49806] AUTH (redacted) 1755714089 423127 [0 127 0 0 1 49806] ping 1755714089 592683 [0 192 168 241 233 42792] XAUTOCLAIM events

Those Redis entries just mean the server’s loop services are busily polling the streams. They don’t guarantee anything made it into Postgres or that you’re actually looking at the same API the UI/clients are talking to. In other words: lots of chatter, no party. What those consumers are - events: the event bus stream - event-persister: reads “events” and writes them to Postgres - task-run-recorder: turns run state changes into events - reactive-triggers: listens to events and fires triggers - actions: orchestrates action events If the UI is empty, the usual culprits are: - API URL mismatch: your workers/flows and your UI are pointed at different APIs - Workspace/tenant mismatch (Cloud) or wrong profile - DB migrations not applied or Postgres unreachable - UI configured to call the wrong API endpoint behind your ingress/proxy Quick, targeted checks Client side (where you run flows/workers) 1) Confirm the API you’re talking to

prefect config view --show-sources

Ensure

PREFECT_API_URL

is your self-hosted Server URL. If this points elsewhere, you’re writing runs to a different API than the UI. 2) Do runs actually exist at that API?

prefect flow-run ls --limit 10

- If you see runs here but not in the UI, the UI is pointed at the wrong API. - If this is empty, you haven’t created runs at this API yet. 3) Are events flowing?

prefect events stream --format text

You should see events for flow/task state changes when you trigger a run. Server side 4) Migrations

prefect server database upgrade

If migrations weren’t applied, the UI will be quite… minimal. 5) Services

prefect server services ls

You should see event-persister, task-run-recorder, etc. If you’re running in Kubernetes, also check pod logs:

kubectl logs deploy/prefect-server -c event-persister
kubectl logs deploy/prefect-server -c task-run-recorder

Look for DB errors or backoffs. Pay special attention to

PREFECT_API_DATABASE_CONNECTION_URL

. UI/API alignment 6) Verify what API the UI is calling - In your browser dev tools, Network tab, filter for

/api/

– do those requests succeed, and what host are they hitting? - In your UI container/Helm values, verify the API URL env (commonly

PREFECT_API_URL

or a

VITE_*

API URL) matches your server API endpoint. A mismatch here = empty UI. Sanity test 7) Minimal flow against your Server - On a machine that can reach your Server, set

PREFECT_API_URL

to your Server’s API. - Run:

from prefect import flow

@flow
def hello():
    return "hi"

if __name__ == "__main__":
    hello()

You should see a flow run in

prefect flow-run ls

and in the UI within a few seconds. A few notes on your Redis entries - XREADGROUP/XAUTOCLAIM from those consumer groups are normal. They indicate the services are polling/claiming messages; they do not confirm persistence into Postgres or that flow/task runs exist. Docs and references - Server services and event persister API reference: Event persister - Managing settings (e.g., PREFECT_API_URL): Manage settings - Settings reference: Settings reference - Workspaces (if you’re on Cloud): Cloud workspaces If you share: - How you’re running Prefect (Server vs Cloud, Helm chart/version) - The

PREFECT_API_URL

from both your worker/flow environment and the UI - The output of

prefect flow-run ls --limit 10

- Any errors from the event-persister/task-run-recorder logs …I can point you to the exact loose screw. I know, thrilling.

Good news: your envs give away the culprit. In your background services pod you never enable the event persister. You’ve set: - PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true - PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=true - PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true - PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true …but not: - PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true When you co-locate services with the API, the default set of services tends to be enabled together and persistence happens — so it “works.” When you split them out, you’ve explicitly enabled some services and accidentally left the persister off — so events are read from Redis but never flushed to Postgres, and the UI remains barren. What to do - Add this to the background services env and redeploy:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true

- Then confirm services are actually running:

prefect server services ls

You should see Event Persister, Task Run Recorder, Scheduler, etc. “running.” Sanity checks (just in case) - Verify both pods use the same image/tag (so they share the same messaging plugin and settings). - Confirm DB connectivity from the services pod. If you’ve never applied migrations in this env:

prefect server database upgrade

- Ensure both API and services agree on Redis settings: - PREFECT_MESSAGING_BROKER=prefect_redis.messaging - PREFECT_REDIS_MESSAGING_HOST/PORT/PASSWORD/DB - If you want to be thorough, from the services pod:

redis-cli -h redis-master.prefect.svc.cluster.local -a redispassword -n 0 XINFO GROUPS events

You should see the consumer groups created by the event persister and task-run recorder. References - Settings reference (service enable flags): Settings reference - Event Persister service API ref: Event persister Flip that one flag and your “inconsistent” matrix should become consistent, in the satisfying way that makes everyone else yawn.

tldr i had to redeploy a thing

if you're still encountering issues i think we could codify this thread as a discussion so we can investigate any rough edges and others can discover it more easily

@Marvin I was able to trace the lifecycle of a task through the redis logs and redis is tracking the task successfully and all 3 instances of the API server and the background services are acknowledging each state change from Pending > Running > Completed. However, I found that the task_run is not updated until 15 minutes afterwards in my Postgres db, and that is what the UI must be reading from. Why is there a 15 minute delay and what service is writing the task state to Postgres specifically?