Short answer: in self‑hosted Prefect, automations run inside the Prefect server process. Both “Send a notification” and “Call a webhook” actions read secrets (your webhook URL) from the server’s database and must decrypt them with the server’s encryption key. If that key is missing or doesn’t match what was used when the action/block was saved, the action fails before any HTTP request is made. In Prefect 3.6.21 those failures are not well surfaced, so it can look “silent”. This exactly matches your symptoms: trigger fires, curl works, no network blocks, but no Slack message and no clear logs. Why this explains each attempt - “Send a notification” with Prefect’s SlackWebhook block: the block document contains your webhook URL as an encrypted secret → if the server can’t decrypt, the action aborts before sending. - “Send a notification” with prefect‑slack Slack Incoming Webhook block: same story; it’s still a block with encrypted fields. Also, this block’s class must be importable in the server image. - “Call a webhook” with the URL typed into the action: that URL is stored as a secret on the automation and decrypted at runtime → it fails the same way if the encryption key is wrong/missing. What to check and fix 1) Ensure the server encryption key is configured consistently - All Prefect server processes (API/web, events/automations worker, any replicas) must share the same Fernet key via the env var: - PREFECT_SERVER_ENCRYPTION_KEY - Do not mix settings (e.g., some processes set the env var, others don’t), and do not rotate it without recreating blocks/automations; otherwise previously saved secrets become unreadable. Generate a key and set it

# Generate a Fernet key (44-char base64 string)
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

# Example ECS task definition env for every server task/container:
PREFECT_SERVER_ENCRYPTION_KEY=YOUR_GENERATED_FERNET_KEY

Notes: - There’s a deprecated alias ORION_ENCRYPTION_KEY; stick with PREFECT_SERVER_ENCRYPTION_KEY. - If you previously ran without the env var, Prefect may have auto-generated and stored a key in the DB. Mixing an env‑var key on one server process with the DB‑stored key used by another will break decryption. Pick one approach and make it consistent everywhere. Docs and references - Self-hosted guide: Self‑hosted Prefect 3 - Security settings (context): Security settings - Encryption utilities reference (how the key is read/used): prefect.server.utilities.encryption 2) After setting the key, restart the server services and re-test - Re-save one of your notification blocks or the automation (to ensure it’s stored with the now-stable key), then trigger a flow to generate a Completed/Failed event. - If you had created blocks/automations while the key was inconsistent, recreate those so they’re encrypted with the correct key. 3) Ensure the plugin is installed in the server image (only if you use the prefect‑slack block) - The automations engine loads the block class server‑side. Make sure the Prefect server container has prefect‑slack installed (not just your worker image).

pip show prefect-slack
# or bake it into the server image / ECS task definition

Using the built‑in

prefect.blocks.notifications.SlackWebhook

does not require the plugin. 4) Enable more verbose logging temporarily on the server - 3.6.21 does not always surface action exceptions well. Raising log level can help catch template/decrypt errors. - In your server config or env:

PREFECT_LOGGING_LEVEL=DEBUG

Then reproduce and check server logs. Optional sanity checks

- From a machine that talks to the same network path as the Prefect server (ideally from inside the server container), curl the Slack webhook to verify egress from that environment, not just from your laptop:

curl -v -X POST -H 'Content-type: application/json' \
  --data '{"text":"Prefect test"}' <https://hooks.slack.com/services/>...

If you want, I can help you verify the key state. A few quick answers would help: - Is PREFECT_SERVER_ENCRYPTION_KEY set on all Prefect server containers/tasks? If so, was it added after you created these blocks/automations? - Is prefect‑slack installed in the Prefect server image (not only workers)? - Are you running multiple server replicas or separate services (API and events/automations) that could have diverging env vars? Once the encryption key is consistent, these Slack notifications typically start working immediately.