<@ULVA73B9P> why I am seeing `Finished in state Cr...
# ask-marvin
a
@Marvin why I am seeing
Finished in state Crashed
in the logs
m
thought for 98.3 seconds
Finished in state Crashed
usually means your flow/task did not fail because of a normal exception in your code — it was interrupted by the runtime or infrastructure. In Prefect 3.x: - Failed = your flow/task raised a normal exception - Crashed = the process was interrupted externally or hit a system-level issue Common causes: - container/pod was killed - worker shut down while the run was executing - OOM / memory pressure - timeout or termination signal -
KeyboardInterrupt
/
sys.exit()
- loss of connectivity to the Prefect API in some cases Prefect maps things like these to a
Crashed
state: - interrupt signals - runtime cancellation - termination signals - some network/timeout errors A good first step is to check the state message in the run details — it often says something more specific, such as: -
Execution was aborted by a termination signal
-
Execution was cancelled by the runtime environment
-
Request to ... failed
A few troubleshooting steps: 1. Check worker/infrastructure logs - Kubernetes: pod events / pod logs - Docker: container exit reason - local worker: worker process logs 2. Look for resource/time limits - OOMKilled - pod eviction - container/job timeout - autoscaling shutdowns 3. Check whether the worker restarted or was stopped - a worker disappearing mid-run often leads to
Crashed
4. Enable debug logging if needed
Copy code
prefect config set PREFECT_LOGGING_LEVEL=DEBUG
Relevant references: - Prefect exceptions: CrashedRun - Troubleshooting Prefect Cloud - Prefect state handling source If you want, paste: - the full state message - where you’re running it (local / Docker / Kubernetes / ECS / etc.) - any worker logs around the crash and I can help narrow down the exact cause.
a
@Marvin it shows
was cancelled by the runtime environment
m
thought for 7.9 seconds
That message means Prefect received an async cancellation from the execution environment, so it marked the run as Crashed instead of Failed. Most likely causes: - the worker process was shutting down - the container/pod was stopped - the infrastructure running the flow was interrupted - the run exceeded some platform timeout - the process was cancelled by the orchestration layer So this usually points to an environment/runtime issue, not a bug in your flow code. What to check next 1. Worker logs around the same timestamp - look for shutdown/restart messages 2. Infrastructure events - Kubernetes: pod eviction, restart,
OOMKilled
, job timeout - Docker: container stopped/restarted - VM/server: process supervisor restart or machine interruption 3. Deployment/work pool settings - any job timeout, TTL, or platform-level execution limits 4. Resource pressure - memory/CPU exhaustion can trigger cancellation indirectly Rule of thumb - If your code threw an exception, you'd usually see Failed - If the environment pulled the rug out from under the run, you'll often see Crashed: was cancelled by the runtime environment If you're on Kubernetes, the most useful thing is usually: - pod status - pod events - worker logs - exit reason If you want, tell me where the flow is running (Kubernetes, Docker, ECS, local worker, etc.) and I can give you the exact places to inspect.
a
@Marvin I have docker pool and it is up for last 7 days
m
thought for 6.5 seconds
If you’re using a Docker work pool and it’s been up for 7 days,
was cancelled by the runtime environment
most often means the flow-run container was stopped/cancelled while it was still running — even if the worker container itself stayed up. So the key distinction is: - worker/container for the pool is up - the individual flow run container may still have been interrupted Things to check in a Docker setup: 1. Did the flow run container exit unexpectedly? Run:
Copy code
bash
docker ps -a
Look for the container created for that flow run and check its exit status. 2. Inspect that container
Copy code
bash
docker inspect <container_id>
Look at: -
State.ExitCode
-
State.OOMKilled
-
State.Error
- start/finish timestamps 3. Check container logs
Copy code
bash
docker logs <container_id>
If the logs cut off abruptly, that supports runtime cancellation / termination. 4. Check whether Docker restarted, pruned, or removed containers Sometimes housekeeping, daemon restarts, or host-level interruptions can stop the run container without killing the long-lived worker. 5. Check host resource pressure Especially memory pressure: - low RAM - OOM killer - disk pressure - Docker daemon restart 6. Check whether the run was manually cancelled If something external requested cancellation, the runtime may surface it this way too. A very common pattern with Docker pools The worker submits a short-lived container for each flow run. The worker can remain healthy for days, while one specific run container gets: - killed - removed - OOM’d - timed out - disconnected So “worker has been up for 7 days” does not rule out a runtime cancellation. If you want, run these and paste the output for the affected run container:
Copy code
bash
docker ps -a
docker inspect <container_id>
docker logs <container_id>
If you don’t know the container id, I can help you find it from the flow run name/time.