Tom Han
10/21/2025, 8:30 PM20:09:27.883 | INFO | Flow run 'platinum-duck' - 2025-10-21 20:09:27 | INFO | sleap_nn.predict:run_inference:260 | Started tracking at: 2025-10-21 20:09:27.883246
20:10:25.077 | INFO | Flow run 'platinum-duck' - 2025-10-21 20:10:25 | INFO | sleap_nn.predict:run_inference:284 | Running tracking on 180000 frames...
20:15:54.021 | WARNING | prefect.events.clients - Unable to connect to '<ws://10.1.37.163:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: [Errno 111] Connect call failed ('10.1.37.163', 4200). Set PREFECT_DEBUG_MODE=1 to see the full error.
Would those flow still run? They should have a good connection now.Marvin
10/21/2025, 8:30 PMMarvin
10/21/2025, 8:36 PMprefect flow-run ls --state RUNNING
prefect flow-run ls --state COMPLETED --limit 50
- Inspect a specific run if you have its ID:
prefect flow-run inspect <flow-run-id>
- View logs captured by the API for a run (note: gaps may exist from the downtime):
prefect flow-run logs <flow-run-id> --tail
If you see a run that you know finished but it’s still showing RUNNING, share a run ID and I can help dig in. In some cases, you may cancel and re-run if the final state didn’t persist.
Given your setup (self-hosted on a Mac mini with SQLite, ~20 workers), a move to Postgres should improve API stability under load, which will also reduce these telemetry warnings.
If this warning pops up again, setting PREFECT_DEBUG_MODE=1 on the worker will provide the full error for easier diagnosis.Tom Han
10/21/2025, 8:37 PMMarvin
10/21/2025, 8:37 PMMarvin
10/21/2025, 8:40 PMprefect flow-run ls --state CRASHED --limit 200
prefect flow-run inspect <flow-run-id>
prefect flow-run logs <flow-run-id> --tail
- Recovery: rerun those flows (trigger the deployment again or use your existing retry/automation policies). There isn’t a safe way to “revive” a CRASHED run to RUNNING/COMPLETED after the fact.
How to avoid this next time
- Pause or widen the “missing heartbeat → CRASHED” automation during maintenance or restarts:
prefect automations ls
prefect automations pause <automation-id>
# later
prefect automations resume <automation-id>
- Increase the heartbeat-missing threshold and/or add conditions (e.g., only for certain tags/work pools) so transient API outages don’t prematurely crash runs.
- Optionally add an automation to auto-retry or re-trigger the deployment when a run is CRASHED for “heartbeat missing.”
References
- States: States
- Automations: Automations
- Events/logging behavior: Events client and Logging to API
If you share one affected flow-run ID, I can confirm its state history and whether it hit the heartbeat automation. Would you like help tuning that automation’s threshold so restarts don’t cause false CRASHED states?Tom Han
10/21/2025, 8:46 PMMarvin
10/21/2025, 8:46 PMMarvin
10/21/2025, 8:50 PM@flow calls it makes (subflows) will still execute locally. Whether they’re tracked in the UI depends on API availability at the moment each subflow starts:
- If the API was reachable when the subflow started, it should have its own flow run recorded and progress independently of the parent’s state.
- If the API was unreachable at that moment, the subflow may still run but not be fully tracked (you may see missing events/logs or a missing subflow run record).
Important nuances
- CRASHED is terminal in the API for that parent run; it won’t “come back.” But it doesn’t kill already-running code.
- Subflows are independent flow runs; they don’t require the parent to be in RUNNING state to execute/report. They’ll attempt to create their own run records and transition states on their own.
- If the worker process actually died, then of course the subflow code won’t be called.
How to check what happened
- Look for subflow runs around the outage window by name:
prefect flow-run ls --flow-name <subflow_flow_name> --limit 100 --state COMPLETED
prefect flow-run ls --flow-name <subflow_flow_name> --limit 100 --state RUNNING
- Inspect specific runs for history/logs:
prefect flow-run inspect <flow-run-id>
prefect flow-run logs <flow-run-id> --tail
References
- States (CRASHED is terminal): States
- Automations overview (for your heartbeat rule): Automations
If you share the parent run ID and the subflow name, I can help verify which subflows were created and their final states.Tom Han
10/21/2025, 8:51 PMMarvin
10/21/2025, 8:51 PMMarvin
10/21/2025, 8:52 PMfrom uuid import uuid4
from prefect.deployments import run_deployment
# inside your flow
child = run_deployment(
name="my-flow/my-deployment",
parameters={"foo": "bar"},
idempotency_key="parent-<parent-run-id>-my-deployment", # make this deterministic per attempt
tags=["triggered-by:parent-flow"],
as_subflow=True, # default; keeps parent/child linkage in the UI
)
- Optionally add a small retry around run_deployment to handle temporary API hiccups.
How to confirm they went through
- List recent runs for the child flow/deployment:
prefect flow-run ls --flow-name <child-flow-name> --limit 100
- Inspect/logs for a specific child run:
prefect flow-run inspect <child-flow-run-id>
prefect flow-run logs <child-flow-run-id> --tail
Docs
- run_deployment reference: run_deployment
If you have any automations that cancel on parent CRASHED, those could stop the child; otherwise, the child deployment runs should proceed independently once created. If you share a parent run ID and the deployment name, I can help verify which child runs were created.