<@ULVA73B9P> I restarted my server while a few wor...
# ask-marvin
t
@Marvin I restarted my server while a few workers are still running, and some of them had this error message:
Copy code
20:09:27.883 | INFO    | Flow run 'platinum-duck' - 2025-10-21 20:09:27 | INFO | sleap_nn.predict:run_inference:260 | Started tracking at: 2025-10-21 20:09:27.883246
20:10:25.077 | INFO    | Flow run 'platinum-duck' - 2025-10-21 20:10:25 | INFO | sleap_nn.predict:run_inference:284 | Running tracking on 180000 frames...
20:15:54.021 | WARNING | prefect.events.clients - Unable to connect to '<ws://10.1.37.163:4200/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: [Errno 111] Connect call failed ('10.1.37.163', 4200). Set PREFECT_DEBUG_MODE=1 to see the full error.
Would those flow still run? They should have a good connection now.
m
thought for 327.2 seconds
Short answer: yes — those flows should keep running. That warning is from the events websocket; it’s used for telemetry (events/logs), not for executing your flow code. A websocket disconnect will not stop the worker from executing the flow. What’s impacted during the outage - Events/telemetry: Some events and logs may not make it to the server while the websocket/API is down. They aren’t backfilled; once the connection is healthy, new events/logs resume. See Events client and Logging to API. - Run state reporting: State transitions are sent over HTTP. If the API was unavailable at the exact moment a state change was reported, that update might have failed. The flow still runs locally; when connectivity returns, subsequent transitions will be reported normally. In some cases a run that finished during the outage may show an outdated state in the UI. See States. How to verify things are healthy now - List currently running/completed flows:
Copy code
prefect flow-run ls --state RUNNING
prefect flow-run ls --state COMPLETED --limit 50
- Inspect a specific run if you have its ID:
Copy code
prefect flow-run inspect <flow-run-id>
- View logs captured by the API for a run (note: gaps may exist from the downtime):
Copy code
prefect flow-run logs <flow-run-id> --tail
If you see a run that you know finished but it’s still showing RUNNING, share a run ID and I can help dig in. In some cases, you may cancel and re-run if the final state didn’t persist. Given your setup (self-hosted on a Mac mini with SQLite, ~20 workers), a move to Postgres should improve API stability under load, which will also reduce these telemetry warnings. If this warning pops up again, setting
PREFECT_DEBUG_MODE=1
on the worker will provide the full error for easier diagnosis.
t
@Marvin I have switched to postgres already, there’s a error with not enough file descriptor so i had to restart server with a bigger ulimit. I also have a workflow that set missing heartbeat to crashed so i am not sure if those flow can come back to life
m
thought for 137.9 seconds
Great move on Postgres and bumping ulimit — that’ll help stability under load. Given you have an automation that marks runs CRASHED when a heartbeat is missing: any runs that were flipped to CRASHED during the server outage will not “come back to life.” In Prefect 3.x, CRASHED is a terminal state. Your workers likely kept executing the code locally, but when they later tried to report new states (e.g., COMPLETED) the API would reject those transitions because the run was already terminal. You may see gaps in logs and “state transition rejected” messages around that window. What you can do now - Identify affected runs:
Copy code
prefect flow-run ls --state CRASHED --limit 200
prefect flow-run inspect <flow-run-id>
prefect flow-run logs <flow-run-id> --tail
- Recovery: rerun those flows (trigger the deployment again or use your existing retry/automation policies). There isn’t a safe way to “revive” a CRASHED run to RUNNING/COMPLETED after the fact. How to avoid this next time - Pause or widen the “missing heartbeat → CRASHED” automation during maintenance or restarts:
Copy code
prefect automations ls
prefect automations pause <automation-id>
# later
prefect automations resume <automation-id>
- Increase the heartbeat-missing threshold and/or add conditions (e.g., only for certain tags/work pools) so transient API outages don’t prematurely crash runs. - Optionally add an automation to auto-retry or re-trigger the deployment when a run is CRASHED for “heartbeat missing.” References - States: States - Automations: Automations - Events/logging behavior: Events client and Logging to API If you share one affected flow-run ID, I can confirm its state history and whether it hit the heartbeat automation. Would you like help tuning that automation’s threshold so restarts don’t cause false CRASHED states?
t
@Marvin So would subflows triggered by a flow marked as crashed still go through?
m
thought for 269.3 seconds
Short answer: yes. A parent being marked CRASHED in the API does not stop the Python process. If the parent flow code is still running, any
@flow
calls it makes (subflows) will still execute locally. Whether they’re tracked in the UI depends on API availability at the moment each subflow starts: - If the API was reachable when the subflow started, it should have its own flow run recorded and progress independently of the parent’s state. - If the API was unreachable at that moment, the subflow may still run but not be fully tracked (you may see missing events/logs or a missing subflow run record). Important nuances - CRASHED is terminal in the API for that parent run; it won’t “come back.” But it doesn’t kill already-running code. - Subflows are independent flow runs; they don’t require the parent to be in RUNNING state to execute/report. They’ll attempt to create their own run records and transition states on their own. - If the worker process actually died, then of course the subflow code won’t be called. How to check what happened - Look for subflow runs around the outage window by name:
Copy code
prefect flow-run ls --flow-name <subflow_flow_name> --limit 100 --state COMPLETED
prefect flow-run ls --flow-name <subflow_flow_name> --limit 100 --state RUNNING
- Inspect specific runs for history/logs:
Copy code
prefect flow-run inspect <flow-run-id>
prefect flow-run logs <flow-run-id> --tail
References - States (CRASHED is terminal): States - Automations overview (for your heartbeat rule): Automations If you share the parent run ID and the subflow name, I can help verify which subflows were created and their final states.
t
@Marvin It's a subflow triggered by a deployment, it is running a deploymeny
m
thought for 63.4 seconds
Got it — you’re calling run_deployment from inside a flow (subflow pattern). - Yes: if the parent flow process was still running, calls to run_deployment will still create and run the target deployment as long as the API was reachable at the moment of the call. The parent being CRASHED in the API does not block this. - No automatic cascade cancel: Prefect does not automatically cancel child runs when a parent is CRASHED (unless you have an automation that does so). The child deployment run is independent and will be picked up by workers normally. A few tips to make this more robust - Use an idempotency key so if you retry after a transient API error you don’t accidentally create duplicates:
Copy code
from uuid import uuid4
from prefect.deployments import run_deployment

# inside your flow
child = run_deployment(
    name="my-flow/my-deployment",
    parameters={"foo": "bar"},
    idempotency_key="parent-<parent-run-id>-my-deployment",  # make this deterministic per attempt
    tags=["triggered-by:parent-flow"],
    as_subflow=True,  # default; keeps parent/child linkage in the UI
)
- Optionally add a small retry around run_deployment to handle temporary API hiccups. How to confirm they went through - List recent runs for the child flow/deployment:
Copy code
prefect flow-run ls --flow-name <child-flow-name> --limit 100
- Inspect/logs for a specific child run:
Copy code
prefect flow-run inspect <child-flow-run-id>
prefect flow-run logs <child-flow-run-id> --tail
Docs - run_deployment reference: run_deployment If you have any automations that cancel on parent CRASHED, those could stop the child; otherwise, the child deployment runs should proceed independently once created. If you share a parent run ID and the deployment name, I can help verify which child runs were created.