https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am running on-prem server and worke...
# ask-marvin
r
@Marvin I am running on-prem server and worker, both version 3.7.1. When flow is running, Prefect UI do not show any tasks in run UI page, but in logs I see : 
Executing task 'resource_pre_task' for task run 'resource_pre_task-3be'...
Why ?
m
✅ thought for 151.7 seconds
Most likely: your task runs are executing, but the server is not recording the task-run events that power the UI’s task list. In Prefect 3.7.x, the UI task list for a flow run depends on server-side event processing, not just worker logs. So this log: 
Executing task 'resource_pre_task' for task run 'resource_pre_task-3be'...
only proves the task is running in the engine. It does not guarantee the server persisted that task run for the UI. Most likely causes 1. 
TaskRunRecorder
service is disabled or unhealthy - This is the server service that turns task events into visible task runs in the UI. - Setting to check: - 
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
- or 
PREFECT_API_SERVICES_TASK_RUN_RECORDER_ENABLED=true
2. Event persistence is disabled/broken - The task recorder depends on event persistence. - Check: - 
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
- or 
PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED=true
3. Worker/server event path is misconfigured - If the worker can submit flow/task state updates enough to run work, but event emission/persistence is not functioning, you can see logs without task rows in UI. 4. Server logs likely contain the real clue - Look for errors mentioning: - 
task_run_recorder
- 
event_persister
- 
events
- database write failures What to check first On the server:
Copy code
bash
prefect config view
Look for values related to: - 
server.services.task_run_recorder.enabled
- 
server.services.event_persister.enabled
Also inspect server logs for those services. If you run Prefect server in Docker, also check env vars on the server container:
Copy code
bash
env | grep PREFECT_SERVER_SERVICES
env | grep PREFECT_API_SERVICES
On the worker, verify it points to the right API:
Copy code
bash
echo $PREFECT_API_URL
Why this happens technically In Prefect 3.7.x, task visibility in the flow run UI depends on task state-change events being: 1. emitted by the worker/client, 2. persisted by the server event system, 3. consumed by the 
TaskRunRecorder
. If that pipeline breaks, tasks still execute, and you still get task logs, but the UI shows no tasks. Relevant docs - Prefect settings: https://docs.prefect.io/latest/guides/settings/ - Prefect self-hosted/server docs: https://docs.prefect.io/latest/manage/self-host/ If you want, I can help you with a step-by-step diagnostic checklist for Docker / Kubernetes / bare-metal, depending on how your on-prem server is deployed.
r
@Marvin PREFECT_SERVER_SERVICES_* settings are not changed, they have default values. I am using deployment in on-prem openshift with 3 API pods and one background service pod. Last message on background service is : 093001.097 | INFO | prefect.prefect_redis.messaging - Consumer event-persister-prefect-background-6dc7d5d6c5-sl8fz-1-1ba6ce47 reconnected to Redis successfully My flow was run at around 13:39. And I do not see any erros in API pods. Where to look for errors ?
m
✅ thought for 171.9 seconds
In your setup, the background services pod is the main place to look. With self-hosted Prefect 3.x, the path is roughly: 
worker emits task-run events -> Redis -> background services consume -> DB -> UI reads from DB
So if tasks run but do not appear in the UI, the failure is usually in one of these places: 1. background services pod logs 2. Redis stream / consumer lag 3. database rows missing for 
task_run
/ task-run events Where to look first ### 1. Background services pod logs This is the most important place. You want logs from the time of the flow run around 
13:39
, not only startup logs. Search for: - 
task_run_recorder
- 
event_persister
- 
Dropping event
- 
Error flushing
- 
IntegrityError
- 
queue full
- 
retry
- 
exception
Example:
Copy code
bash
oc logs <background-pod> --since=4h
Then filter:
Copy code
bash
oc logs <background-pod> --since=4h | grep -Ei "task_run|event|drop|flush|retry|error|exception|integrity"
What matters is whether there are any messages after 09:30. If the last meaningful background log is from 09:30 but your flow ran at 13:39, that is suspicious by itself. ### 2. Verify the background pod is actually still healthy Even if the pod is "running", the service loop may be stuck or disconnected. Check pod status and restarts:
Copy code
bash
oc get pod <background-pod> -o wide
oc describe pod <background-pod>
Look for: - restarts - OOMKilled - probe failures - connection issues to Redis or Postgres ### 3. Check whether task-run events reached the database This is the fastest way to narrow it down. If you can query Postgres, check:
Copy code
sql
SELECT id, name, flow_run_id, created
FROM task_run
WHERE created > NOW() - INTERVAL '2 hours'
ORDER BY created DESC;
And task-run events:
Copy code
sql
SELECT event, occurred
FROM event
WHERE event LIKE 'prefect.task-run%'
  AND occurred > NOW() - INTERVAL '2 hours'
ORDER BY occurred DESC
LIMIT 50;
Interpretation: - events exist, but no 
task_run
rows -> problem is likely 
TaskRunRecorder
- neither events nor task_run rows exist -> problem is earlier, likely event publishing/persistence - task_run rows exist -> then this may be more of a UI/query issue ### 4. Check Redis stream / consumer groups Since you are using Redis messaging, you should verify consumers are actually consuming. Look for consumer groups on the 
events
stream, especially: - 
event-persister
- 
task-run-recorder
If you have Redis access:
Copy code
bash
redis-cli XINFO GROUPS events
redis-cli XLEN events
You want to see whether: - the groups exist - pending entries are growing - the stream is filling but not being consumed If 
event-persister
is alive but 
task-run-recorder
is absent or stalled, that would explain your symptom. Important detail for your deployment With 3 API pods + 1 background pod, the API pods usually won’t show the relevant errors for this issue. The API pods can be perfectly clean while the background services pod is failing to: - consume Redis events - persist events - record task runs So absence of API errors does not rule this out. What I’d check next, in order 1. Background pod logs around 
13:39
2. Whether there are any logs after 
09:30
3. 
task_run
rows in Postgres 4. 
event
rows for 
prefect.task-run%
5. Redis 
XINFO GROUPS events
Very likely outcome If I had to guess from what you shared: - the worker is executing tasks correctly - the API is mostly fine - the background service is likely not processing task-run events correctly after that Redis reconnect If you want, I can help you with a precise diagnostic script/checklist for OpenShift using: - 
oc logs
- 
oc exec
- Postgres SQL - Redis checks and make it copy-pasteable for your environment.
r
@Marvin What I have found: 1. Last message in background pod is from 9:30 2. events are writen in PG, but task_run are not writen 3. I have no access to redis (auth is needed) 4. There are no errors in API log What can I do ?
m
✅ thought for 27.9 seconds
That narrows it down a lot. If: 1. background pod has no logs after 09:30 2. 
event
rows are being written to Postgres 3. 
task_run
rows are not being written 4. API logs are clean then the most likely issue is: 
TaskRunRecorder
in the background services pod is not running correctly anymore, or is stuck, even though 
EventPersister
is/was working. Since 
event
rows are present, event emission and event persistence are working. The broken step is specifically: 
event table / Redis events -> TaskRunRecorder -> task_run table
What to do now ### 1. Restart the background services pod This is the first practical action. Because: - the last background log is from 09:30 - your flow ran at 13:39 - no 
task_run
rows are being produced the background service process is very likely hung, stalled, or disconnected in a way that is not surfacing clearly in logs. On OpenShift:
Copy code
bash
oc delete pod <background-pod>
Let the deployment recreate it, then watch logs immediately:
Copy code
bash
oc logs -f <new-background-pod>
You want to see fresh startup logs and then activity when a new flow runs. ### 2. Re-run a small test flow and watch whether 
task_run
rows appear After restart, run a tiny flow with 1-2 tasks and check:
Copy code
sql
SELECT id, name, flow_run_id, created
FROM task_run
WHERE created > NOW() - INTERVAL '10 minutes'
ORDER BY created DESC;
If rows appear after restart, that strongly confirms the background service got stuck. ### 3. Increase logging on the background services pod Right now, the biggest problem is lack of visibility. If you can change env vars for that deployment, temporarily set:
Copy code
bash
PREFECT_LOGGING_LEVEL=DEBUG
PREFECT_SERVER_LOGGING_LEVEL=DEBUG
If your chart/manifests support logger-specific config, also try:
Copy code
bash
PREFECT_LOGGING_LOGGERS__PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER__LEVEL=DEBUG
PREFECT_LOGGING_LOGGERS__PREFECT_SERVER_EVENTS_SERVICES_EVENT_PERSISTER__LEVEL=DEBUG
Then redeploy/restart the background pod. This should make it easier to see whether: - 
TaskRunRecorder
starts - it consumes events - it retries/fails writes - it stops logging unexpectedly ### 4. Confirm the background pod is actually running the task recorder service If you can exec into the pod, inspect env and process behavior:
Copy code
bash
oc exec -it <background-pod> -- env | grep PREFECT
You want to confirm there is no accidental setting disabling it, even if you did not configure it intentionally. Look especially for:
Copy code
bash
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED
PREFECT_API_SERVICES_TASK_RUN_RECORDER_ENABLED
They should not be 
false
. ### 5. Check for pod resource problems A silent stall can also be caused by resource pressure. Inspect:
Copy code
bash
oc describe pod <background-pod>
Look for: - OOMKilled - restarts - liveness/readiness failures - node/network events Also check current/recent resource usage if available:
Copy code
bash
oc adm top pod <background-pod>
If CPU or memory is starved, the service may appear alive but stop processing. ### 6. Scale/recreate the background deployment cleanly If restart helps only temporarily, recreate the deployment rollout:
Copy code
bash
oc rollout restart deployment/<background-deployment-name>
If you can, keep only one background services replica unless you specifically intend to scale it and have validated that behavior. Why this is the right conclusion Since 
event
rows are in Postgres: - worker -> API/events path is fine - 
EventPersister
worked at least enough to persist events Since 
task_run
rows are missing: - the failure is after event persistence or in the recorder path - that points directly at 
TaskRunRecorder
Since background logs stopped hours before your run: - the recorder service likely stopped processing long before the flow started What I would do immediately 1. restart background pod 2. watch fresh logs 3. run a trivial flow 4. query 
task_run
5. if fixed, add debug logging and investigate why the pod became silent If restart does not fix it Then I would suspect one of these: - 
TaskRunRecorder
service loop is not starting at all - it is blocked on Redis consumption - it is failing on DB writes without sufficient logging - there is a bug in 3.7.1 affecting the recorder in your topology At that point, the next best step is to capture: - background pod startup logs - one test flow timestamp - counts from 
event
and 
task_run
- pod description / restart count and compare behavior after restart. If you want, I can help you build a minimal OpenShift runbook with exact 
oc
commands and SQL checks to verify whether the restart resolved it.
r
@Marvin I have restarted background pod. It does not help, but I can see logs :
Copy code
15:03:42.468 | INFO | prefect.server.services.perpetual_services - Scheduling perpetual service: monitor_worker_health
15:03:42.473 | INFO | prefect.server.services.perpetual_services - Scheduling perpetual service: monitor_late_runs
15:03:42.482 | INFO | prefect.server.services.perpetual_services - Scheduling perpetual service: monitor_expired_pauses
15:03:42.487 | INFO | prefect.server.services.perpetual_services - Scheduling perpetual service: schedule_deployments
15:03:42.492 | INFO | prefect.server.services.perpetual_services - Scheduling perpetual service: schedule_recent_deployments
15:03:42.497 | INFO | prefect.server.services.perpetual_services - Scheduling perpetual service: monitor_expired_leases
15:03:42.503 | INFO | prefect.server.services.perpetual_services - Scheduling perpetual service: evaluate_proactive_triggers_periodic
15:03:42.514 | INFO | prefect.server.services.perpetual_services - Perpetual services: 11 enabled, 1 disabled, 12 total
15:03:42.634 | INFO | prefect.server.services.task_run_recorder - Creating TaskRunRecorder consumer with batch size 1 and flush every 5 seconds
15:03:42.639 | INFO | prefect.server.events.services.triggers - ReactiveTriggers starting with unique consumer name: reactive-triggers-prefect-background-f99577fff-vtjnv-1-db88fb4c
15:03:42.645 | INFO | prefect.server.events.actions - Starting action message handler
/home/prefect/venv/lib64/python3.12/site-packages/prefect_redis/client.py:65: UserWarning: Redis URL is set; the following fields are ignored: db, host, password, port, ssl, username
warnings.warn(
15:03:42.651 | INFO | prefect.server.events.triggers - Starting automation change listener
15:03:42.776 | INFO | prefect.server.utilities.postgres_listener - Successfully established raw asyncpg connection for LISTEN/NOTIFY to <http://sk21csp59v.dctest.slsp.sk/prefect_poc|sk21csp59v.dctest.slsp.sk/prefect_poc>
15:03:42.777 | INFO | prefect.server.events.triggers - Listening for automation changes on prefect_automation_changes
15:03:43.051 | INFO | prefect.server.services.scheduler - Scheduled 0 runs.
m
✅ thought for 34.0 seconds
Thanks — this is very helpful. Your logs show that 
TaskRunRecorder
is starting:
Copy code
INFO | prefect.server.services.task_run_recorder - Creating TaskRunRecorder consumer with batch size 1 and flush every 5 seconds
So this is not a simple “service disabled” problem. Given everything you’ve found, the failure is now most likely one of these: 1. 
TaskRunRecorder
is consuming nothing from Redis 2. it is consuming events, but filtering them out 3. it is trying to write task runs and failing silently / too quietly 4. the task-run events being produced are not in the format the recorder expects And since: - 
event
rows are written to Postgres - 
task_run
rows are not - 
TaskRunRecorder
starts - no clear errors are logged the strongest hypothesis is: the recorder is seeing events, but skipping them because they do not match its filters / expected schema. In Prefect 3.x, 
TaskRunRecorder
only records task-run events if they meet certain conditions, including the orchestration metadata on the event. What to do next ### 1. Inspect the actual persisted task-run events in Postgres This is now the key step. You need to inspect one of the 
prefect.task-run%
events from the 
event
table and compare its contents. Run something like:
Copy code
sql
SELECT
  event,
  occurred,
  resource,
  related,
  payload
FROM event
WHERE event LIKE 'prefect.task-run%'
ORDER BY occurred DESC
LIMIT 5;
What you want to check in particular inside 
resource
/ event data: - does it contain a task-run resource id like: - 
prefect.task-run.<uuid>
- does it contain: - 
"prefect.orchestration": "client"
That field is important because the recorder only accepts client-orchestrated task-run events. If that field is missing or different, the recorder will ignore the event. ### 2. Compare one event from a failing run If possible, filter by the time of your test flow:
Copy code
sql
SELECT
  id,
  event,
  occurred,
  resource,
  related
FROM event
WHERE event LIKE 'prefect.task-run%'
  AND occurred > NOW() - INTERVAL '30 minutes'
ORDER BY occurred DESC;
If you paste one redacted sample of 
resource
and 
related
, I can help interpret it. ### 3. Turn on debug logging specifically for the recorder Right now you can see startup, but not whether it is processing events. Temporarily set on the background pod:
Copy code
bash
PREFECT_LOGGING_LEVEL=DEBUG
PREFECT_SERVER_LOGGING_LEVEL=DEBUG
PREFECT_LOGGING_LOGGERS__PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER__LEVEL=DEBUG
Then restart background pod and run a small test flow. You want to see whether it logs receipt/processing of task-run events. ### 4. Check whether flow-run events are recorded but task-run events are skipped You already know 
event
rows exist, but verify whether they include task-run events specifically:
Copy code
sql
SELECT event, COUNT(*)
FROM event
WHERE occurred > NOW() - INTERVAL '1 hour'
GROUP BY event
ORDER BY COUNT(*) DESC;
If you see flow-run events but no 
prefect.task-run.*
, then the worker is not emitting task-run events the way the recorder expects. If you do see 
prefect.task-run.running
, 
prefect.task-run.completed
, etc., then we focus on event contents. ### 5. Check for schema/content mismatches For one task-run event, inspect: - 
event
- 
resource
- 
related
- 
payload
Especially whether: - task run id is present - flow run id is present in related resources - orchestration metadata is present Why I think this is likely a filtering/content issue Because: - the event pipeline is alive enough to persist events - the recorder starts normally - there are no obvious DB failures - 
task_run
rows remain empty That pattern often means: events exist, but the recorder declines to record them. What would confirm it If your 
event
row looks roughly like this, it should work:
Copy code
json
{
  "prefect.resource.id": "prefect.task-run.<uuid>",
  "prefect.resource.name": "resource_pre_task",
  "prefect.orchestration": "client"
}
If 
prefect.orchestration
is absent, null, or not 
client
, that is a likely root cause. Best next step Please run this query and share one sample row, redacted if needed:
Copy code
sql
SELECT
  event,
  resource,
  related,
  payload
FROM event
WHERE event LIKE 'prefect.task-run%'
ORDER BY occurred DESC
LIMIT 1;
That will probably tell us exactly why 
TaskRunRecorder
is not writing 
task_run
.
r
@Marvin result :
Copy code
{
  "prefect.run-count": "1",
  "prefect.state-name": "Completed",
  "prefect.state-type": "COMPLETED",
  "prefect.resource.id": "prefect.task-run.019eacea-3513-709c-ab87-1ce23bba77aa",
  "prefect.orchestration": "client",
  "prefect.resource.name": "resource_deliver_task-48b",
  "prefect.state-message": "",
  "prefect.state-timestamp": "2026-06-09T15:04:50.717279+00:00"
}
m
✅ thought for 232.4 seconds
That event sample is good news. It confirms at least this much: - 
prefect.resource.id
is correct - 
prefect.orchestration
is 
client
- the event is a real task-run event shape So the recorder should not be filtering it out for the two most common reasons. That means the problem is probably one of these next-level causes: 1. the event payload is incomplete, even though the 
resource
field looks fine 2. the event references a 
flow_run
that the recorder cannot resolve/write against 3. the recorder is hitting DB write failures/retries that are not obvious at current log level 4. the event is persisted in 
event
, but the recorder is reading from Redis and not from the DB rows you inspected That last point is important: 
TaskRunRecorder
consumes the event stream, not the 
event
table. So it is possible for: - 
EventPersister
to write 
event
rows to Postgres - while 
TaskRunRecorder
is not successfully consuming/processing the same stream Because both services consume separately. What to check next ### 1. Inspect 
payload
, not just 
resource
The recorder also needs event payload data. Please run:
Copy code
sql
SELECT
  event,
  resource,
  related,
  payload
FROM event
WHERE event LIKE 'prefect.task-run%'
ORDER BY occurred DESC
LIMIT 1;
I specifically want to know whether 
payload
contains: - 
task_run
- 
validated_state
If either is missing, that would explain why no 
task_run
row is created. ### 2. Check whether the event has a related flow run The recorder works best when the event includes the related flow run. In the 
related
field, look for something like:
Copy code
json
{
  "prefect.resource.id": "prefect.flow-run.<uuid>",
  "prefect.resource.role": "flow-run"
}
If that relation is missing or points to a non-existent flow run, inserts can fail. ### 3. Verify the flow run itself exists in DB Take the flow-run id from the UI or the related event resource and query:
Copy code
sql
SELECT id, state_type, created
FROM flow_run
WHERE id = '<flow-run-uuid>';
If the task event references a flow run that is not present, 
TaskRunRecorder
can fail to insert task runs. ### 4. Turn on debug logging for the background pod At this point, I strongly recommend it. Use:
Copy code
bash
PREFECT_LOGGING_LEVEL=DEBUG
PREFECT_SERVER_LOGGING_LEVEL=DEBUG
PREFECT_LOGGING_LOGGERS__PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER__LEVEL=DEBUG
Then restart the background pod and run a tiny test flow. What we want to catch: - whether 
TaskRunRecorder
is actually receiving task-run events from Redis - whether it logs retries/failures when writing ### 5. Check whether any 
task_run
rows are ever created Run:
Copy code
sql
SELECT COUNT(*), MAX(created)
FROM task_run;
If the count is zero for the whole environment, that suggests a systemic recorder problem. If older rows exist but new ones don’t, that suggests a regression/stall. Most likely diagnosis now Based on what you shared, my current ranking is: 1. 
TaskRunRecorder
is not consuming the Redis stream correctly, even though it starts 2. payload/related fields are incomplete for recorder persistence 3. DB write failures involving flow-run linkage / task-run upsert logic Because your 
resource
field already looks valid. Most useful next thing you can share Please paste the result of:
Copy code
sql
SELECT
  resource,
  related,
  payload
FROM event
WHERE event LIKE 'prefect.task-run%'
ORDER BY occurred DESC
LIMIT 1;
That should let us tell whether the event itself is incomplete, or whether the problem is almost certainly in the recorder’s Redis consumption / DB write path. If you want, after that I can help you build a very targeted SQL checklist to distinguish: - bad event content - missing flow-run reference - recorder not consuming Redis - recorder DB insert failure
r
@Marvin I will change log level to debug. But before that, can you explain how worker comunicate to server (API or background) - can you write schema for whole communication during flow execution, includinng worker, server, background and database ?
m
✅ thought for 48.2 seconds
Yes — here’s the practical picture for a self-hosted Prefect 3.x setup like yours. High-level rule The worker talks to the API server, not directly to the background services pod. The background services pod is mostly for asynchronous server-side processing: - scheduling - automations - late-run monitoring - event consumption - task-run recording - etc. So the path is not: 
worker -> background
It is more like: 
worker <-> API -> DB / Redis <- background
--- Main components in your deployment - Worker - polls for scheduled work from Prefect API - runs flow code - reports states/logs/events - API pods - serve REST API and UI backend - read/write core orchestration data to Postgres - publish/handle event traffic - Background services pod - runs server services that consume from messaging and do async processing - e.g. 
TaskRunRecorder
, 
EventPersister
, scheduler, triggers - Postgres - source of truth for flow runs, deployments, states, logs, events, etc. - Redis - messaging bus / stream backend for Prefect server event services in your deployment --- Whole communication flow during flow execution Here is a simplified schema:
Copy code
text
                    +----------------------+
                    |     Prefect UI       |
                    |  (browser frontend)  |
                    +----------+-----------+
                               |
                               | HTTP
                               v
                   +-------------------------+
                   |       API pods          |
                   |  FastAPI / orchestration|
                   +----+---------------+----+
                        |               |
              SQL read/write            | publish/consume event-related traffic
                        |               |
                        v               v
                 +------------+    +-----------+
                 | Postgres   |    |   Redis   |
                 | DB         |    | streams   |
                 +------------+    +-----+-----+
                                         |
                                         |
                                         v
                          +-------------------------------+
                          |   Background services pod     |
                          | - TaskRunRecorder             |
                          | - EventPersister              |
                          | - Scheduler                   |
                          | - Triggers                    |
                          | - Late-run monitors           |
                          +-------------------------------+

Worker side:

+-------------+
|   Worker    |
+------+------+ 
       |
       | poll for work / submit state / logs / events
       v
+-------------------------+
|       API pods          |
+-------------------------+
--- Step-by-step lifecycle of a deployed flow run ## 1. Worker polls for work The worker periodically asks the API for flow runs to execute.
Copy code
text
Worker -> API pods
This includes: - polling a work pool / work queue - claiming scheduled flow runs The API reads from Postgres to determine what should run.
Copy code
text
API pods <-> Postgres
--- ## 2. API returns a flow run to the worker Once work is available, the API responds with flow-run metadata.
Copy code
text
API pods -> Worker
The worker then starts flow execution locally in its runtime environment. --- ## 3. Worker starts the flow run As the flow starts, the worker/runtime reports flow-run state changes back to the API. Examples: - 
Pending
- 
Running
- 
Completed
- 
Failed
Copy code
text
Worker -> API pods -> Postgres
These are core orchestration writes. --- ## 4. Worker executes tasks inside the flow Inside the Python runtime, tasks are created/executed by the engine. During this process, the runtime can generate: - task-run states - task logs - events Important distinction:
### A. Logs Task/flow logs generally go through the API and are stored in DB.
Copy code
text
Worker -> API pods -> Postgres
That is why you can often see logs even when task rows are missing. ### B. Events The runtime also emits Prefect events such as: - 
prefect.flow-run.*
- 
prefect.task-run.*
These are part of the event system. Conceptually:
Copy code
text
Worker/runtime -> API/event endpoint -> Redis stream
Then background services consume them. --- ## 5. EventPersister consumes events One background service, 
EventPersister
, consumes events from Redis and writes them to the 
event
table in Postgres.
Copy code
text
Redis -> Background(EventPersister) -> Postgres.event
This is why you found 
event
rows in PG. --- ## 6. TaskRunRecorder consumes task-run events Another background service, 
TaskRunRecorder
, also consumes task-run events from the event stream and creates/updates task-run records in Postgres.
Copy code
text
Redis -> Background(TaskRunRecorder) -> Postgres.task_run / task_run_state
This step is exactly where your issue seems to be. If this step breaks: - tasks still execute - logs still appear - events may still appear in 
event
- but 
task_run
rows never show in UI --- ## 7. UI reads from API The UI never talks directly to DB or background pod. It calls the API, and the API reads from Postgres.
Copy code
text
UI -> API pods -> Postgres
So if 
task_run
rows are missing in Postgres, the UI cannot show tasks. --- Detailed communication map by data type ## A. Deployment / scheduling
Copy code
text
User/UI -> API -> Postgres
Scheduler(background) -> Postgres
Worker -> API -> Postgres
## B. Flow run state
Copy code
text
Worker -> API -> Postgres(flow_run, flow_run_state)
## C. Task execution logs
Copy code
text
Worker -> API -> Postgres(log)
## D. Prefect events
Copy code
text
Worker/runtime -> API/event ingestion -> Redis
Redis -> EventPersister(background) -> Postgres(event)
Redis -> TaskRunRecorder(background) -> Postgres(task_run, task_run_state)
Redis -> other background consumers
--- Why logs can exist while tasks are missing This is the confusing part, but it makes sense with the architecture. These are separate paths:
Copy code
text
Task log visibility:
Worker -> API -> DB(log table)

Task row visibility in flow UI:
Worker -> event pipeline -> Redis -> TaskRunRecorder -> DB(task_run)
So a broken 
TaskRunRecorder
path gives exactly your symptom: - logs say task executed - UI task list is empty --- Concrete schema for your case With your OpenShift deployment: ```text +------------------+ | Browser / UI | +--------+---------+ | v +---------------------------+ | API pod 1 / 2 / 3 | | - REST API | | - UI backend | +-----+----------------+----+ | | | SQL | event ingest / publish v v +-----------+ +----------+ | Postgres | | Redis | +-----------+ +----+-----+ | v +----------------------------------+ | Background services pod | |----------------------------------| | TaskRunRecorder | | EventPersister | | Scheduler | | Triggers |
| Late run monitors | +----------------------------------+ +----------------------+ | Worker | | runs flow code | +----------+-----------+ | | API calls v +-------------+ | API pods | +-------------+
Copy code
---

*What happens for one task in a running flow*

Example task `resource_pre_task`:

```text
1. Worker starts flow
2. Worker executes task resource_pre_task
3. Worker logs:
   "Executing task 'resource_pre_task' ..."
4. Log goes to API -> Postgres(log)
5. Runtime emits task-run event
6. Event enters Redis stream
7. EventPersister writes it to Postgres(event)
8. TaskRunRecorder should write Postgres(task_run, task_run_state)
9. UI asks API for flow run details
10. API reads Postgres(task_run)
11. If no row exists, UI shows no tasks
--- Important consequence Because you already confirmed: - 
event
rows exist - 
task_run
rows do not we can narrow your problem to:
Copy code
text
Redis/event stream -> TaskRunRecorder -> Postgres.task_run
The worker-to-API communication is probably fine. --- Short version - Worker communicates with API, not background directly - API writes orchestration data to Postgres - Events go through Redis - Background services consume Redis - TaskRunRecorder writes 
task_run
rows - UI reads from API, which reads Postgres --- If you want, I can next draw this as a sequence diagram for a single flow run with: - deployment pickup - flow start - task execution - logs - event persistence - task-run recording - UI rendering That may make it even easier to debug where your pipeline is breaking.
r
@Marvin One clarification : event table is in fact called 
events
, right ? Please draw sequence diagram, it will be helpful.
m
✅ thought for 27.8 seconds
Yes — thanks for the correction. In Prefect’s Postgres DB, the table is 
events
 (plural), not 
event
. Here’s a sequence diagram for a typical deployed flow run in your setup. ```text Participant legend: - UI = Prefect UI in browser - API = one of your Prefect API pods - Worker = Prefect worker - Redis = messaging/event stream - Background = background services pod - PG = Postgres ┌────┐ ┌─────┐ ┌────────┐ ┌──────┐ ┌────────────┐ ┌─────┐ │ UI │ │ API │ │ Worker │ │ Redis│ │ Background │ │ PG │ └─┬──┘ └──┬──┘ └───┬────┘ └──┬───┘ └─────┬──────┘ └──┬──┘ │ │ │ │ │ │ │ create deployment / run │ │ │ │ │────────────>│ │ │ │ │ │ │ write flow/deployment/run │ │ │ │ │────────────────────────────────────────────────────────────────>│ │ │ │ │ │ │ │ │ worker polls for work │ │ │ │ │<──────────────│ │ │ │ │ │ read scheduled work │ │ │ │ │────────────────────────────────────────────────────────────────>│ │ │ return flow run to execute │ │ │ │ │──────────────>│ │ │ │ │ │ │ │ │ │ │ │ │ start flow │ │ │ │ │ │ update flow state RUNNING │ │ │ │<──────────────│ │ │ │ │ │────────────────────────────────────────────────────────────────>│ │ │ │ │ │ │ │ │ │ execute task │ │ │ │ │ │ "resource_pre_task" │ │ │ │ │ │ │ │ │ │ │ send logs │ │ │ │ │<──────────────│ │ │ │ │ │ write log rows --->│ │ │────────────────────────────────────────────────────────────────>│ │ │ │ │ │ │ │ │ │ emit Prefect event │ │ │ │<──────────────│ │ │ │ │ │ publish event │ │ │ │ │ │──────────────────────────────>│ │ │ │ │ │ │ │ │ │ │ │ │ EventPersister consumes │ │ │ │ │<───────────────────│ │ │ │ │ │ │ write events │ │ │ │ │ │─────────────>│ │ │ │ │ │ │ │ │ │ │ TaskRunRecorder consumes │ │ │ │ │<───────────────────│ │ │ │ │ │ │ write task_run,
│ │ │ │ │ task_run_state│ │ │ │ │ │─────────────>│ │ │ │ │ │ │ │ UI opens flow run page │ │ │ │ │────────────>│ │ │ │ │ │ │ read flow run + tasks + logs │ │ │ │ │────────────────────────────────────────────────────────────────>│ │ │ return data │ │ │ │ │<────────────│ │ │ │ │ │ │ │ │ │ │
Copy code
*Key takeaway for your issue*

Your current evidence suggests this part works:

```text
Worker -> API -> Redis -> Background(EventPersister) -> PG.events
and this part is failing:
Copy code
text
Worker -> API -> Redis -> Background(TaskRunRecorder) -> PG.task_run / PG.task_run_state
So in your case: - worker execution: works - API communication: works - logs: work - 
events
table writes: work - task-run persistence: does not work --- Even shorter split by table ### Flow states
Copy code
text
Worker -> API -> PG.flow_run / flow_run_state
### Logs
Copy code
text
Worker -> API -> PG.log
### Events
Copy code
text
Worker -> API -> Redis -> Background(EventPersister) -> PG.events
### Task rows visible in UI
Copy code
text
Worker -> API -> Redis -> Background(TaskRunRecorder) -> PG.task_run / task_run_state
--- Why UI can show logs but no tasks Because the UI reads different data sources from Postgres: - logs come from 
log
- task list comes from 
task_run
and 
task_run_state
So if 
task_run
is empty, the run page can still show logs. --- If you want, I can next make a failure-point diagram that marks exactly where each symptom appears, like: - “logs visible, tasks missing” - “events present, no task_run” - “API clean, background suspicious” That would map directly onto your current incident.
a
@Robert Szomolanyi Hey Robert, I am facing the same issue in 
3.7.4
. I have another thread just above yours. After some debugging, I opened up an issue to learn more. Does it match with your issue?
r
@Anish S After turning DEBUG mode on, task are displayed in UI. Now is everythink OK. I think that my background job was running with older version of Prefect and when we turned on DEBUG logging, image was rebuild with correct Prefect verzion (3.7.1).
Open in Slack
PreviousNext
Powered by