Thanks for the report — that combo (no block types showing in the “Add block” modal + notifications failing to submit) usually points to the UI not being able to talk to the API correctly or the API being mid-upgrade (DB schema mismatch), rather than an isolated UI bug. A few targeted checks and quick fixes (assuming Prefect 3.x — please confirm your version): Quick things to try - Hard refresh the UI (Shift+Reload) or open in a private window to rule out a cached UI after an upgrade. - Check the browser dev tools → Network tab when you: - Click “Add block” (should call GET /api/block_types) - Click “Create” on a notification (should call POST /api/notifications) - Note the HTTP status codes (403, 422, 500) and any error JSON Server health and migrations - Verify the API is healthy:

curl -sSf http://<your-api>/api/health
  curl -sSf http://<your-api>/api/version

- If you recently upgraded Prefect or the DB, run migrations (then restart the API/UI):

prefect server database upgrade -y

Docs: Server CLI • Server concepts System block types (what populates the “Add block” list) - If the block list is empty, system block types may not be registered or the endpoint is failing. - Manually (re)register core blocks:

prefect blocks register -m prefect.blocks.notifications
    prefect blocks register -m prefect.blocks.system
    prefect blocks register -m prefect.blocks.webhook

- Or call the API to (re)install system blocks:

curl -X POST http://<your-api>/api/block_types/install_system_block_types

Docs: Blocks • Install system block types API Reverse proxy and UI → API config - If you’re behind a reverse proxy (nginx/Traefik/ingress), ensure the UI knows how to reach the API: - Set

PREFECT_UI_API_URL

to the API URL as seen by the browser (e.g., http://<your-host>/api, not an internal container name) - Ensure the proxy preserves headers and cookies (important for CSRF, see below) Docs: Self-hosted (advanced) CSRF errors on POST (common cause for “Failed to submit notification”) - A 403 on POST endpoints (notifications, blocks) typically means the CSRF token didn’t make it through. - In the Network tab, check that the POST has an

X-CSRF-Token

header and that the API responds without 403. - Make sure your proxy does not strip `Set-Cookie`/custom headers and that both

PREFECT_UI_URL

and

PREFECT_UI_API_URL

point to the externally accessible addresses. Docs: Security settings • CSRF middleware reference: API middleware Version compatibility - If the UI or CLI was updated but the server wasn’t (or vice versa), you can get 422/500s on certain endpoints. - Compare

/api/version

with your UI/CLI version (

prefect version

) and keep them aligned on 3.x. Docs: Server concepts If you can share a bit more, I can pinpoint this faster: - Prefect version(s):

prefect version

and

/api/version

- How you’re hosting (Docker Compose/Helm/k8s? reverse proxy in front?) - Any recent changes (upgraded Prefect image, changed proxy config, restored DB, etc.) - The exact HTTP status codes and responses from: - GET

/api/block_types

when opening “Add block” - POST

/api/notifications

when creating a notification - Relevant API logs around those times If you want a minimal “fix it now” sequence to try: 1) Run DB migrations and restart:

prefect server database upgrade -y

2) Reinstall system block types: ```

curl -X POST http://<your-api>/api/block_types/install_system_block_types ``` 3) Confirm GET

/api/block_types

returns items, then hard refresh the UI. 4) If POST still fails for notifications and you see 403, adjust proxy to preserve headers/cookies and set correct

PREFECT_UI_API_URL

. Links for reference: - Server health: Health check - Blocks: Blocks - Install system blocks: API - Self-hosted server/CLI: Server CLI • Concepts - Security/CSRF: Security settings • API middleware Happy to help walk through logs/screenshots if you can share the specifics above.

Great question. Short answer: run them wherever you can reach your Prefect API, and for the Prefect CLI, run it in the same environment/image as your server so it uses the same config. Here are concrete examples for common setups: - If you’re using Docker Compose - Run DB migrations inside the API container:

docker compose exec <api-service-name> prefect server database upgrade -y

- Register system blocks (same place):

docker compose exec <api-service-name> prefect blocks register -m prefect.blocks.notifications
    docker compose exec <api-service-name> prefect blocks register -m prefect.blocks.system
    docker compose exec <api-service-name> prefect blocks register -m prefect.blocks.webhook

- Or run a one-off container with the same image/env:

docker run --rm --network <compose_network> \
      -e PREFECT_API_URL=http://<api-service-name>:4200/api \
      prefecthq/prefect:<your-version> \
      prefect server database upgrade -y

- curl can be run on your workstation against the public URL (if exposed) or from a container on the same network:

curl -X POST http://<your-public-host-or-api-service>:<port>/api/block_types/install_system_block_types

- If you’re on Kubernetes - Exec into the API pod (or a toolbox pod using the same image/env):

kubectl exec -it <api-pod> -- prefect server database upgrade -y
    kubectl exec -it <api-pod> -- prefect blocks register -m prefect.blocks.notifications

- curl from anywhere that can reach the service (public ingress or cluster-internal):

curl -X POST http://<api-ingress-host>/api/block_types/install_system_block_types

or

kubectl run -it curl --rm --image=curlimages/curl -- \
      curl -X POST http://<api-service>.<namespace>.svc.cluster.local:4200/api/block_types/install_system_block_types

- If you installed Prefect directly on a host/VM - Run commands in the same environment/venv where Prefect is installed, pointing the CLI at your API:

export PREFECT_API_URL=http://<api-host>:4200/api
    prefect server database upgrade -y
    prefect blocks register -m prefect.blocks.notifications
    curl -X POST http://<api-host>:4200/api/block_types/install_system_block_types

Notes: - Make sure the CLI can reach your API (

PREFECT_API_URL

set to your server’s API URL). In 3.x self-hosted, the UI uses

PREFECT_UI_API_URL

, but the CLI uses

PREFECT_API_URL

. - If you enabled server auth, set

PREFECT_API_AUTH_STRING="username:password"

in the environment where you run the Prefect CLI. - Keep the CLI and server on the same Prefect 3.x version to avoid incompatibilities. Tell me how you’re hosting Prefect (Compose, k8s, bare metal) and your API URL, and I’ll tailor the exact commands for your setup.

Got it — Coolify runs your Prefect server in a Docker container behind its proxy. Here’s exactly where to run things and what to set. Step 1 — Run the Prefect CLI inside your Prefect container - In the Coolify UI, open your Prefect app → Exec/Console into the running container. - Run these inside that container (they’ll use the same DB/env as your server):

prefect server database upgrade -y

  prefect block register -m prefect.blocks.notifications
  prefect block register -m prefect.blocks.system
  prefect block register -m prefect.blocks.webhook

Notes: - The command is

prefect block register

(singular).

prefect blocks register

also works, but I recommend the singular form shown above. - If you’ve enabled server auth, ensure the container has

PREFECT_SERVER__API__AUTH_STRING

set (see Step 3), otherwise the CLI may fail to authenticate to your own API. Step 2 — Call the API from your laptop (optional but handy) - You can hit your public domain through Coolify’s proxy:

curl -X POST https://<your-prefect-domain>/api/block_types/install_system_block_types
  curl -sSf https://<your-prefect-domain>/api/health
  curl -sSf https://<your-prefect-domain>/api/version
  curl -s https://<your-prefect-domain>/api/block_types

If you get non-200s here, share the status/body and we can diagnose (403 suggests CSRF/proxy headers, 422/500 suggests version or DB). Step 3 — Verify Coolify environment variables on the Prefect service - In your Prefect app settings in Coolify, set: -

PREFECT_UI_API_URL=https://<your-prefect-domain>/api

-

PREFECT_UI_URL=https://<your-prefect-domain>

- If you enabled server auth:

PREFECT_SERVER__API__AUTH_STRING=username:password

- Redeploy/restart the service after changing env vars so the UI picks them up. - These ensure the UI (running inside the server) knows how to reach the API through Coolify’s public URL. Step 4 — Retest - Hard refresh the UI (Shift+Reload). - Try “Add Block” again — it should list block types. - Try creating a notification — if it fails, open the browser Network tab and check: - The POST to

/api/notifications

status (403 = CSRF/proxy issue; 500 = backend/DB; 422 = version mismatch) - We can fix based on that code. If you prefer to run the CLI from your local machine instead of inside the container: - Set your local env so the CLI can reach your server:

export PREFECT_API_URL=https://<your-prefect-domain>/api
  # if server auth is enabled:
  export PREFECT_API_AUTH_STRING="username:password"
  prefect server database upgrade -y
  prefect block register -m prefect.blocks.notifications

- Keeping your local Prefect CLI version aligned with the server version avoids incompatibilities. Helpful docs: - Server CLI: Server CLI - Server concepts: Server concepts - Blocks: Blocks - Install system blocks (API): API - Security/CSRF: Security settings If you share: - Your public Prefect domain, - Whether server auth is enabled, - And the HTTP status you see for GET /api/block_types and POST /api/notifications, I can tailor the exact next fix for your Coolify setup.