Sebastian S
11/25/2025, 4:53 PMError running periodically process followers taskIt's followed by
Periodically process followers task cancelled successfullyAnd after this, we see no more task_run_recorder logs or tasks created in UI/DB. Wondering if it hit an issue and silently died?
Marvin
11/25/2025, 4:53 PMMarvin
11/25/2025, 4:56 PMtask_run_recorder logs and no new tasks appear in the UI/DB, that usually means the Task Run Recorder consumer (or the whole server process) exited or got stuck—often due to an underlying DB/broker/connectivity issue. The cancellation message is a symptom of the shutdown, not the cause.
Context on the feature: it was added as a periodic task to handle “lost followers” here: PR #15100.
A few next checks to narrow this down:
- Grab the full traceback right after “Error running periodically process followers task” — that will usually point to the root cause (e.g., DB connection, lock, or query error).
- Look for any “service starting/restarting” messages after the cancellation log; if the process restarted, you should see the periodic task “starting” log again.
- Confirm your DB is healthy (connections available, no locks/timeouts) around that timestamp.
- If you can, bump logging to DEBUG for prefect.server.services.task_run_recorder to get more detail, then reproduce.
- What Prefect version are you on? There have been fixes in this area; upgrading to the latest 3.x patch is recommended.
Could you share:
- Your Prefect version and whether this is Prefect Cloud or OSS Server
- The full traceback around that error
- Any adjacent DB/broker logs around the same time
With that, I can help pinpoint whether this is a transient DB/broker hiccup, a service crash, or something we should escalate.