Thanks for the detailed report — super helpful. Short answer - “Server type: ephemeral” is almost certainly not the root cause here given that your flow runs and subflows are visible in the UI. That message usually just means the CLI you ran didn’t have PREFECT_API_URL set and fell back to an ephemeral client for that command. - In Prefect 3.x, both direct task calls (

task()

) and submitted calls (

task.submit()

) emit task state events and should result in TaskRun rows being persisted and visible in the UI when the server’s Task Run Recorder service is running. - What you’re seeing (0 task runs in UI and none returned by

/api/task_runs/filter

) strongly suggests the Task Run Recorder service in your self‑hosted API is disabled or not running/consuming events. What’s likely happening - In Prefect 3, TaskRun ORM rows are created asynchronously by the “Task Run Recorder” server service which consumes task state change events emitted by the client/worker. If that service is disabled or failing, your flow runs will still be created (by the orchestration API) and logs will show up, but task runs will not be recorded — resulting in “0 task runs” for every flow in the UI and empty results from

/api/task_runs/filter

. - The fact that you’ve seen a “Crashed” task row appear once via

submit()

but not for direct calls is consistent with an event consumer that is not consistently processing events; it’s not expected behavior when the recorder is healthy. Direct answers to your questions 1) Is “Server type: ephemeral” the root cause? - No. If your flows and subflows appear in the UI backed by Postgres, you are not actually using an ephemeral server for orchestration. That CLI message is a red herring unless your worker also lacks PREFECT_API_URL (which would prevent flow runs from appearing at all — not your case). 2) Does

task()

vs

task.submit()

affect whether task runs are registered? - No. In Prefect 3.x both paths emit task state events; both should result in TaskRun records in the DB. The difference you’re seeing points to the server not recording task runs, not to a behavioral difference you need to code around. 3) Is there additional config needed for persistence on self‑hosted? - Ensure the Task Run Recorder service is enabled and healthy on your API server. The key setting is: -

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

(default is true, but worth verifying) - Also ensure your API server is correctly configured to use Postgres (which it is, since flow runs are persisted), and optionally disable ephemeral fallback globally to avoid surprises: -

PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=false

(optional, belts-and-suspenders) How to verify and fix - Check your API server (Docker) environment: - Confirm you’re running the Prefect 3.6.21 image/tag on the API container (not just the worker). - Confirm

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED

is either unset (defaults to true) or explicitly set to

true

. - Check API server logs for the recorder service: - Look for startup logs indicating the task run recorder started (logger usually contains “task_run_recorder”). - If you can, temporarily increase log level for

prefect.server.services.task_run_recorder

to DEBUG and watch for messages about consuming and persisting task run events. - Quick functional check: - Run a tiny flow with one task, then wait a couple seconds after completion and query the API:

from prefect import flow, task, get_client
    import asyncio, time

    @task
    def t():
        return 42

    @flow
    def f():
        return t()

    fr = f()
    time.sleep(2)  # give the recorder a moment to persist

    async def check(fr_id):
        async with get_client() as c:
            trs = await c.read_task_runs(flow_run_filter={"id": {"any_": [fr_id]}})
            print("Task runs for flow:", trs)

    asyncio.run(check(fr.model_dump()["id"]))

- If that still returns an empty list, the recorder is not persisting. - If you’re using Docker Compose/Helm, please share the API container env/config (or confirm the above flags). Also share any “task_run_recorder” errors from the API logs — that will pinpoint the issue quickly. Extra notes - There can be a small lag (hundreds of ms) between flow completion and task run visibility, but not persistent “always zero” as you’re seeing. - The UI’s “Task Runs” tab uses the same

/api/task_runs/filter

you tested. If that endpoint returns empty for a specific

flow_run_id

, it’s a server-side persistence/recording issue — not a UI filter quirk. If you can paste your API container image tag and its environment variables (redact secrets) plus a snippet of the API logs around startup showing whether the task run recorder starts, I can help you zero in on the exact misconfiguration.

Great follow-ups — and those log lines help. 1) Logs vs. persistence - The “Task run … | INFO” lines are emitted locally by the SDK’s task engine when a task enters/exits states. They do not, by themselves, guarantee that a task run was recorded in the API database. - Recording TaskRuns in Prefect 3 happens asynchronously via the event pipeline: - The client/worker emits task-run state-change events. - The server’s Task Run Recorder service consumes those events and writes/updates TaskRun rows. - If the client cannot deliver events (e.g., bad PREFECT_API_URL, WebSocket blocked, network interruption) or the server-side recorder is disabled/unhealthy, you’ll still see those INFO logs while TaskRuns will be missing from the DB/UI. 2) Process management in your setup (Process worker) - The Process worker launches each flow run in a separate Python subprocess. - Inside that flow-run process, tasks are executed by the flow’s task runner: - Default is ConcurrentTaskRunner (thread pool). So: - Direct call

task()

runs inline on the main thread (still emits events). -

task.submit()

runs in a worker thread via the task runner (also emits events). - There is no new OS process per task unless you explicitly use a task runner that uses processes (e.g., Dask/Ray/multiprocess integrations). - Subflows called from within a parent flow run execute in the same flow-run process by default (but they still create distinct flow run records in the API). 3) Debugging task recording end-to-end Client/worker-side (ensure events are being sent) - Raise logging to DEBUG in your worker environment: -

PREFECT_LOGGING_LEVEL=DEBUG

- Optional for deeper internals:

PREFECT_INTERNAL_LOGGING_LEVEL=DEBUG

- What to look for in worker logs: - Successful connection to the events stream, e.g., “Opening events stream to …/events/in” / “WebSocket connected” - Event send attempts and acks - Errors like “Failed to connect to events endpoint,” “WebSocket closed,” or retries - Quick connectivity test from the flow environment: - Ensure the flow process has the same

PREFECT_API_URL

the worker advertises. - Run a tiny flow with one task and then wait a second; if the recorder is healthy you should see 1 TaskRun appear. If not, check server logs (below). - If you run a reverse proxy (Nginx/ALB/etc.) in front of the Prefect API, make sure WebSocket upgrades are allowed to the events ingress. The client uses WebSocket at

/events/in

for events by default. A blocked/closed WS path will prevent TaskRuns from being recorded while everything else (REST/logs) can appear to work. - Example Nginx location for events:

location /events/in {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass <http://prefect-api>;
    }

Server-side (verify events are received and recorded) - Enable verbose server logging: -

PREFECT_SERVER_LOGGING_LEVEL=DEBUG

(alias often also supported:

PREFECT_LOGGING_SERVER_LEVEL=DEBUG

) - Ensure the Task Run Recorder is enabled: -

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

(default is true, but verify) - Optionally, enable the EventLogger service to see every event the server receives: -

PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true

- This will print incoming events, including type (e.g.,

prefect.task-run.*

) and resources. It’s very helpful to confirm arrival. - What to look for in server logs: - WebSocket handshake/connection messages for

/events/in

- EventLogger output showing

prefect.task-run.*

events arriving - Task Run Recorder messages indicating it’s consuming and writing task runs (or any warnings like “dropping event …”) - The recorder only persists events where the resource contains `prefect.orchestration: "client"`; that’s what the SDK emits for tasks. API checks you can run

- After a small, known flow completes (1 task), give it ~1–2 seconds and query the API:

curl -s -X POST "$PREFECT_API_URL/api/task_runs/filter" \
    -H "Content-Type: application/json" \
    -d '{"flow_runs":{"id":{"any_":["<FLOW_RUN_ID>"]}}}'

- If this is empty across runs, the recorder likely isn’t persisting or the events aren’t reaching the server. Common root causes that match your symptoms - WebSocket path to

/events/in

blocked by proxy/security group. Flows and logs can work while task events never arrive. - Task Run Recorder disabled or crashing silently in the API container. - Worker process has PREFECT_API_URL set, but the spawned flow-run subprocess does not (less likely since your flow runs persist, but double-check the runtime environment for the flow subprocess). - Server logging level too low to reveal recorder errors; bump to DEBUG to confirm. If you can share: - The API container env (redact secrets): values for

PREFECT_SERVER_LOGGING_LEVEL

,

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED

, and whether

PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED

is set. - Any reverse proxy in front of the API and its config for WebSocket upgrades to

/events/in

. - A short snippet of server logs at startup plus a few seconds while you run a one-task flow. I can then help pinpoint whether events are failing to reach the server or being dropped by the recorder.

Great questions — here’s how the events (task recording) path works and what to check in your VPC setup. 1) Is the /events/in WebSocket only between Client and Server? - Yes. The WebSocket connection is opened by the Prefect SDK process that is executing your flow/tasks (i.e., the flow-run subprocess that your worker launches). It connects directly to the Prefect API’s WebSocket endpoint at

/events/in

. - There is no “agent/worker relay” for events — the worker just launches the flow-run process with the right environment; the flow-run process itself emits events to the API. - On the server side, the API receives those events and then hands them to the internal event pipeline; the Task Run Recorder service consumes them and writes TaskRun rows. 2) VPC with internal IPs, no ALB/NGINX — could there still be an issue? - Yes, a few common ones even without a proxy: - Security Groups/NACLs: Ensure inbound TCP to the API port (e.g., 4200) from the worker/flow hosts is allowed and outbound from workers is allowed. - Docker port publishing: If the API runs in Docker, confirm the host port is published (e.g.,

-p 4200:4200

) and listening on the instance’s interface (not just 127.0.0.1). - Scheme/port mismatch: The SDK derives the WS URL from

PREFECT_API_URL

. If

PREFECT_API_URL

uses

https://...

, the client will attempt

wss://...

to

/events/in

. A self-signed/invalid cert or wrong port will break the WS. - DNS/name resolution: If using a hostname for

PREFECT_API_URL

, verify it resolves to the internal IP you expect from the worker/flow host. - Host firewall (ufw/iptables/SELinux): Can drop long-lived TCP connections. - Without ALB/NGINX, there’s no “upgrade” layer to misconfigure, but a blocked port, wrong scheme, or Docker bind issue will still prevent the WS connection and silently stop task runs from being recorded. 3) Does the WebSocket use the same port as standard API traffic? - Yes. The WebSocket endpoint is served by the same uvicorn/FastAPI app and uses the same host and port as `PREFECT_API_URL`; only the scheme changes: - If

PREFECT_API_URL

is

<http://10.0.1.12:4200/api>

, events go to

<ws://10.0.1.12:4200/events/in>

- If

PREFECT_API_URL

is

<https://internal.domain:443/api>

, events go to

<wss://internal.domain:443/events/in>

Practical checks in your environment Client/worker side - Confirm basic TCP reachability to the API port from the machine running flows:

nc -vz 10.0.1.12 4200

- Run a tiny flow with DEBUG logging to see the events handshake:

export PREFECT_LOGGING_LEVEL=DEBUG
  python - <<'PY'
  from prefect import flow, task
  @task
  def t(): return 1
  @flow
  def f(): return t()
  f()
  PY

- Healthy: you should see logs indicating the events stream was opened/connected. - If you see messages about failing to connect to the events stream/WebSocket, it’s a networking/scheme/port issue. Server side - Enable verbose server logging and the event logger service to confirm arrival of events:

export PREFECT_SERVER_LOGGING_LEVEL=DEBUG
  export PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true
  export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

- You should see logs for incoming

prefect.task-run.*

events during a test run. - If no events appear while a test flow runs, the WS isn’t connecting (or is connecting to a different API instance). - If events arrive but TaskRuns still don’t show up, look for Task Run Recorder warnings/errors — that service is responsible for persisting TaskRuns. Common misconfigurations to double‑check -

PREFECT_API_URL

correctness in the flow-run subprocess environment (same value as the worker uses; must point to your server, not localhost). - HTTP vs HTTPS: if you used

https://

for

PREFECT_API_URL

but the server is actually plain HTTP, the client will try

wss://

and fail the TLS handshake. - Docker port exposure on the API host.

- Security group/NACL rules for the API port between the worker’s subnet and the server instance. If you share your

PREFECT_API_URL

form (redact host if needed, but include scheme and path), the API container port mapping, and whether the server is HTTP or HTTPS, I can help validate the exact WS URL and the fastest next check.

Nice test — that 400 with “missing Sec-WebSocket-Key” proves the endpoint is listening and reachable on that port, which is great. Now we need to confirm whether the flow-run process actually opens that WebSocket and whether the server is receiving and recording the events. Direct answers - Version mismatch: With both SDK and server at 3.6.21, a protocol mismatch is very unlikely to be the culprit. - Force detailed WS failure logs from the SDK: Yes — set

PREFECT_DEBUG_MODE=1

in the flow-run subprocess environment. The events client only includes the full exception text for connection failures when debug mode is on. You can also raise the specific logger for events:

prefect.events.clients

. Next steps to pinpoint the break Client/flow-run side - Ensure these are set in the actual flow-run environment (not just the systemd worker):

PREFECT_API_URL=http://<server>:4200/api
  PREFECT_LOGGING_LEVEL=DEBUG
  PREFECT_INTERNAL_LOGGING_LEVEL=DEBUG
  PREFECT_DEBUG_MODE=1

Tip: Put them in your deployment’s environment or the work pool default environment so the spawned flow subprocess inherits them. - Run a tiny diagnostic flow and crank up the events client logger:

python - <<'PY'
  import logging
  from prefect import flow, task
  from prefect.settings import PREFECT_API_URL

  # Target the events client logger specifically
  logging.getLogger("prefect.events.clients").setLevel(logging.DEBUG)

  @task
  def t():
      from prefect.settings import PREFECT_API_URL
      print("API URL in task:", PREFECT_API_URL.value())
      return 1

  @flow
  def f():
      return t()

  f()
  PY

What you should see: - Logs like “Opening events stream to ws(s)://…/events/in”, “Authenticating…”, “Pinging…” - If it cannot connect after its retries, you’ll see a warning with a suggestion to set PREFECT_DEBUG_MODE=1 (which you already did) and, in debug, the exact exception (e.g., TLS/handshake/timeout). - If you see none of these messages, the process may be using a NullEventsClient — re-check that

PREFECT_API_URL

is visible in the flow-run process. Common client-side causes even when curl “works” - Scheme mismatch: If

PREFECT_API_URL

is

https://…

, the SDK will try

wss://…/events/in

. Make sure the server is actually serving TLS on that port; otherwise use

http://

so it picks

ws://

. - Different environment between worker and flow-run subprocess: Systemd env != deployment/work-pool env. Confirm

PREFECT_API_URL.value()

prints what you expect inside the task (you mentioned it does — good). - Docker publish/bind: If your API is in Docker, ensure it’s bound on 0.0.0.0 and the port is published (e.g.,

-p 4200:4200

). Curl from the worker host reaching the port is a good sign, but make sure the hostname in

PREFECT_API_URL

resolves identically from inside the flow-run process. Server side - Turn on verbose logging and the event logger to confirm arrival:

PREFECT_SERVER_LOGGING_LEVEL=DEBUG
  PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true
  PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

- During a test flow, you should see

prefect.task-run.*

events printed by the EventLogger. - If you see no events at all while the test is running, the WS never connected (or connected to a different endpoint). - If you see events but still zero TaskRuns, look for Task Run Recorder messages/errors — that service is responsible for persisting TaskRuns. - Quick DB/API check after a one-task flow finishes (give it ~1–2s):

curl -s -X POST "$PREFECT_API_URL/api/task_runs/filter" \
    -H "Content-Type: application/json" \
    -d '{"flow_runs":{"id":{"any_":["<FLOW_RUN_ID>"]}}}'

Why Event Logger helps answer “curl works but no TaskRuns” - If Event Logger shows nothing: the client is not connecting (scheme/auth/URL/env issue) even though the port is open.

- If Event Logger shows only flow-run events and not task-run events: tasks may not be emitting (rare) or the flow never calls tasks as tasks (not your case given the INFO logs). - If Event Logger shows task-run events but DB is empty: Task Run Recorder disabled or erroring — check for its logs and ensure

PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true

. Extra sanity checks - Confirm the exact

PREFECT_API_URL

format inside the task (you already did): it should include

/api

, e.g.,

<http://10.x.x.x:4200/api>

. - If you’re using HTTPS on 4200 with a self-signed cert, verify the flow-run process trusts it (wss handshakes will fail otherwise). - No proxy/ALB is good — nothing else needs to manage the upgrade — but still confirm Security Groups/NACLs allow long‑lived TCP connections from worker to server. If you can share: - The exact

PREFECT_API_URL

(scheme + path, host redacted is fine), - A few lines of server logs with Event Logger enabled during a test run, - Any “prefect.events.clients” debug output from the test script above, I can help zero in on whether we’re dealing with a client connection issue or a server-side recording issue.