<@ULVA73B9P> Hi — on a self-hosted Prefect server,...
# ask-marvin
u
@Marvin Hi — on a self-hosted Prefect server, the triggers/actions service appears not to pick up any automation changes made after the server started. I'm seeing two related symptoms that point to the same root cause: 1. Editing an existing automation has no effect. After I change an automation (e.g. switch the Slack notification channel), it keeps firing with the old definition — notifications still go to the previous channel. The action runs, but it uses the pre-edit config. 2. Newly created automations never fire. A brand-new automation doesn't trigger at all on matching events, as if it was never registered with the triggers service. Together this looks like the triggers service is stuck on the set of automation definitions loaded at server startup — edits aren't reloaded (so old config is used) and new ones aren't loaded (so they never fire). Setup - Prefect 3.6.21 (client + server), API 0.8.4, Python 3.12.3 - Self-hosted single-instance server (
prefect server start
), PostgreSQL (AWS RDS), Linux x86_64 - Logging level: DEBUG Other observations - The Slack block works fine manually (
SlackWebhook.load(...).notify(...)
succeeds), so the action target is OK. - Automations are enabled (toggle ON) and configs look correct in the UI. Already ruled out: webhook block (works manually), API URL mismatch (client and server on the same instance), event services (event_persister / triggers / actions / stream_out all enabled). Questions 1. Is there a known issue where the triggers service doesn't reload automation definitions (edits or new ones) without a server restart on self-hosted? 2. Which server log lines confirm whether the triggers service is picking up automation create/update events? (I saw references to an "automation change listener" — what should I look for?) 3. I found #19332 (automation_bucket FK violation on self-hosted Postgres, reported on 3.4.x) — could something similar still affect 3.6.x? 4. Is there a way to force the triggers service to reload automations, or a recommended fix / patch version?
m
thought for 193.7 seconds
What you’re seeing does line up with the triggers service holding stale in-memory automation state. Short answer: 1. Yes, this is consistent with a known class of issues in self-hosted Prefect 3.x 2. The most useful logs are from the automation change listener and the periodic reconciliation path 3. The older FK bug in #19332 sounds different from your symptoms 4. The practical fix is likely *upgrade to a newer 3.x patch*; there is not a supported public “reload automations now” command Based on code-level research, Prefect’s triggers service keeps automations in memory and is supposed to refresh them via: - PostgreSQL
LISTEN/NOTIFY
change events - a periodic reconciliation loop that reloads automations if DB state changed - a full reload after a DB notification connection reconnects Your symptoms suggest one or both of these refresh paths are not working in your version/runtime. What should happen internally On startup, the triggers service does an initial full automation load. After that, automation changes should be picked up by: - a listener for Postgres notifications on automation create/update/delete - a periodic reconciliation check If either path is broken: - edited automations keep using the old config already loaded in memory - new automations never get loaded into the triggers service That matches your report very closely. Answers to your questions 1. Is this a known issue? Yes, broadly yes. I found evidence of fixes in this area after the 3.6.x line for: - automations being missed after the Postgres notification connection is interrupted and re-established - missing/failing notification delivery needing a periodic reconciliation fallback So on
3.6.21
, it is plausible you are hitting behavior that was improved in later 3.x releases. I did not find a GitHub issue with exactly your wording from the quick issue/discussion search, but the source-level behavior and later fixes strongly suggest this is a real, known failure mode. 2. Which logs confirm whether the triggers service is seeing automation changes? Look in server logs for messages around: - startup of the triggers service - startup of the automation change listener - receipt of automation change notifications - periodic evaluation / reconciliation - listener errors and reconnects The most useful log patterns to search for are:
Copy code
bash
automation change listener
Listening for automation changes
Received automation change notification
Loaded .* automations
Running periodic evaluation
Error in automation change listener
Re-established PostgreSQL LISTEN/NOTIFY connection
In a healthy system, after creating or editing an automation, you’d expect to see something like: - listener started - notification received for that automation - automation reloaded, or at least a subsequent “Loaded X automations with Y triggers” Examples of relevant messages found in the code path include:
Copy code
text
Starting automation change listener
Listening for automation changes on prefect_automation_changes
Received automation change notification: created for <automation_id>
Received automation change notification: updated for <automation_id>
Error in automation change listener: <error>. Reconnecting in 10s...
Re-established PostgreSQL LISTEN/NOTIFY connection, reloading all automations to reconcile in-memory state
Loaded X automations with Y triggers
Running periodic evaluation as of <timestamp>
If you never see: -
Received automation change notification...
and/or - periodic evaluation followed by reload behavior then the triggers service likely is not refreshing its in-memory definitions. 3. Could #19332 be related? Probably not directly. The issue you mentioned — #19332 — was about an
automation_bucket
foreign key violation on self-hosted Postgres in 3.4.x.
That sounds more like a persistence/data-integrity failure during automation processing, whereas your symptoms are specifically: - old automation definitions continuing to execute - new definitions not loading at all That points more strongly to automation synchronization/reload problems than to the same FK issue. So: possible same general subsystem, but likely not the same bug. 4. Is there a way to force reload automations, or a recommended fix? There does not appear to be a documented/public CLI or API to tell the triggers service “reload all automations now”. The reliable workaround is: - restart the server / triggers service The better fix is: - upgrade to a newer Prefect 3.x patch release Based on the code research, later fixes were added for: - reloading automations after Postgres listener reconnect - periodically reconciling automations even if notifications are missed So I’d strongly recommend testing on the latest available 3.x release rather than staying on
3.6.21
. You can review current release notes here: Prefect releases If you want, I can also help you compare your
3.6.21
behavior against the latest release notes for automation-related fixes. What I’d do next in your environment 1. Collect server logs while editing/creating an automation Search for:
Copy code
bash
   automation change listener|Listening for automation changes|Received automation change notification|Loaded .* automations|Running periodic evaluation|Error in automation change listener|Re-established PostgreSQL LISTEN/NOTIFY connection
2. Check whether the listener is failing silently or reconnecting repeatedly Especially anything mentioning: - Postgres connection resets - LISTEN/NOTIFY errors - reconnect loops 3. Confirm whether periodic reconciliation is active If you never see periodic evaluation logs at DEBUG, that’s a clue. 4. Restart the server and confirm automations work immediately after restart If restart makes the edited/new automations behave correctly, that’s additional evidence the in-memory automation registry is stale. 5. Upgrade to the latest 3.x patch This is my main recommendation. My best assessment Given your exact symptoms on self-hosted
3.6.21
with Postgres: - yes, this could absolutely be a stale automation registry problem in the triggers service - yes, later Prefect changes appear to address this class of issue - no, I don’t think #19332 is the most likely explanation - no, there’s not a public “force reload automations” control; restart or upgrade are the practical paths If you want, send me: - a small slice of server logs from startup - logs from the moment you edit or create an automation - how you are launching the server (
prefect server start
, container, Helm, etc.) and I can help you pinpoint whether your listener, reconciliation, or Postgres notification path is the one failing.
u
@Marvin Thanks! One question on the upgrade — I'm currently on 3.6.21. Which version should I target for the "reload automations after LISTEN/NOTIFY reconnect" and "periodic reconciliation" fixes? Is there a specific release where those landed, or would you just recommend the latest 3.x patch?
m
thought for 10.7 seconds
I’d recommend at least
3.7.5
, and in practice I’d go with the latest available 3.x patch. Why: - The latest release notes I checked are for
3.7.5
- That release is
<https://github.com/PrefectHQ/prefect/compare/3.7.4...3.7.5|3.7.5>
- Based on the code-level research, the automation-refresh fixes you care about landed after
3.6.x
, and
3.7.5
is the first version I can point to confidently as a safe target from the currently available release stream So my recommendation would be: - Minimum target:
3.7.5
- Preferred target: latest 3.x patch available when you upgrade A couple of notes: - I can’t confidently name the exact first patch number where each of those two fixes first shipped just from the release summary output alone - But I can say
3.6.21
is old enough that upgrading to the current
3.7.x
patch line is the safest move for this issue class Relevant links: - Prefect releases - 3.7.5 changelog If you want, I can do one more pass and try to map those two fixes to the exact PRs / first released version numbers so you can choose the narrowest upgrade target.