Thanks for the details — let’s narrow this down. In Prefect 3.x, Slack alerts from automations depend on server-side services (events → triggers → actions). If your Slack block works when called directly but automations stopped sending, it’s almost always one of: - The server’s background services for automations aren’t running (or were restarted/disabled) - The automation now points at a stale block document (the action references a block by ID; recreating a block with the same name won’t update the automation) - The trigger no longer matches the events being emitted (e.g., wrong states, tags, resources) - A silent server-side error (permission/read-only filesystem, etc.) is preventing actions from running Here’s a concise checklist to find the culprit. 1) Verify server services are running Run this in your Prefect server pod:

prefect server services ls

You should see Triggers and Event Persister enabled. If either is disabled, set env vars and restart the pod:

PREFECT_API_SERVICES_TRIGGERS_ENABLED=true
PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED=true

Also check logs for errors:

kubectl logs -f <your-prefect-server-pod> | grep -i 'automation\|trigger\|action\|slack\|permission'

Look for “Action failed”, “PermissionError”, “referenced block was not found”, etc. 2) Confirm the automation is enabled and what it references List and inspect the automation:

prefect automations ls
prefect automations inspect "<automation-name-or-id>"

Check: -

enabled

is true - The trigger is watching what you expect (failed/crashed/timed out flow runs) - The action references a valid

block_document_id

3) Ensure the referenced block still exists (by ID) Automations store a block_document_id. If you recreated your Slack block, the ID changed. - Get the block ID from the automation’s action - Inspect the block(s):

prefect block ls
prefect block inspect slack-webhook/<your-block-name>

If the automation points at a non-existent/old block ID, re-edit the automation and re-select the Slack block (or recreate the automation). 4) Watch events live while reproducing a failure Open a stream and then make a flow fail/crash:

prefect events stream --format text

You should see: - flow events like

prefect.flow-run.failed

,

prefect.flow-run.crashed

, or

prefect.flow-run.timed-out

- automation events like

prefect.automation.trigger.fired

,

prefect.automation.action.triggered

, and either

prefect.automation.action.executed

or

prefect.automation.action.failed

If you see the flow-run events but no automation events, your trigger isn’t matching; if you see

action.failed

, the payload usually explains why (e.g., block not found). 5) Double‑check your CLI usage for creating automations In Prefect 3.6.3 there is no

prefect automation create

or

prefect automations create

command. The supported subcommands are:

prefect automations ls|inspect|enable|disable|pause|resume|delete

If you’re managing automations from YAML, that must be via the UI or a custom/scripted API call. Please share the exact

automations.yaml

and how you’re applying it so we can confirm the automation was actually created/updated. 6) Cluster specifics to rule out - Multiple server replicas: if you recently scaled up/down or restarted, make sure all replicas share the same Postgres and that Triggers/Event Persister are enabled. Avoid in‑memory event backends in multi‑replica setups. - Read-only containers: if your server pod runs with a read‑only filesystem, check logs for PermissionError around service startup; that can prevent actions from executing. If you can, please share: - Output of

prefect server services ls

- Output of

prefect automations inspect "<name-or-id>"

- The action’s

block_document_id

and the output of

prefect block inspect slack-webhook/<name>

- A short snippet of

prefect events stream --format text

while a flow crashes - Your

automations.yaml

(or a redacted version)

With those, I can pinpoint exactly where it’s breaking and get you back to receiving Slack alerts.