That message means the task did not fail because of an exception in your task code — it was interrupted from outside the task. What it usually means - Prefect marked the task as Crashed, not just Failed - The runtime environment cancelled execution while the task was running - In practice, that often means one of these happened: - the flow run or task run was cancelled - the worker process was shutting down - a Kubernetes pod/container was terminated or evicted - the process hit a timeout or external shutdown signal How to interpret it - Failed usually means your code raised an exception - Crashed usually means the execution environment disappeared or cancelled the run So this log:

text
[ERROR] [prefect.task_runs] Crash detected! Execution was cancelled by the runtime environment.

is basically saying:

text
"This task was stopped by the runtime, not by a normal Python exception in user code."

Common causes to check 1. Was the flow manually cancelled? 2. Did the worker restart or shut down? 3. If running on Kubernetes, was the pod evicted/restarted/terminated? 4. Did you configure a timeout that may have triggered cancellation? 5. Did the parent flow stop, causing child tasks to be cancelled too? What to look at next - the flow run state and surrounding logs - the worker logs - your infrastructure logs: - Kubernetes:

kubectl describe pod ...

- Docker/container platform logs - VM/service supervisor logs Important note Task retries usually help with task code failures, but a crash/cancellation often points to an infrastructure/runtime issue instead. If you want, I can help you narrow it down if you share: - how you're running Prefect (local worker, Docker, Kubernetes, ECS, etc.) - whether the flow run also shows cancelled/crashed - the logs immediately before/after this message Relevant docs: - Prefect states - Prefect workers - Deployments in Prefect 3

I am running on prefect 2.20.25. I have set up activeDeadlineSeconds for k8s job to be terminated after a certain time period. Also the worker has been restarted multiple times. How to confirm which of these 2 events are actually referring to the log message?

@Marvin I am running on prefect 2.20.25. I have set up activeDeadlineSeconds for k8s job to be terminated after a certain time period. Also the worker has been restarted multiple times. How to confirm which of these 2 events are actually referring to the log message?

Yes — you can usually distinguish them by checking the Kubernetes pod/job termination reason, not just the Prefect log. That Prefect log by itself only tells you:

text
the task was cancelled by the runtime

It does not tell you whether the cancellation came from: -

activeDeadlineSeconds

- worker restart / worker disappearance - another infra-level interruption Best way to confirm Check the job pod and job status in Kubernetes for the specific failed run. If

activeDeadlineSeconds

caused it, Kubernetes will usually show: - pod/container termination reason like

DeadlineExceeded

- job condition with reason

DeadlineExceeded

- pod phase often

Failed

If worker restart caused it, you’re more likely to see: - no

DeadlineExceeded

- pod deletion / missing pod / orphaned state - possibly

Evicted

,

NodeLost

, missing exit code, or abrupt shutdown behavior - Prefect-side symptoms like inability to determine exit code / pod disappearing unexpectedly Commands to run 1. Inspect the job:

bash
kubectl get job <job-name> -o yaml

Look for conditions like:

yaml
reason: DeadlineExceeded
type: Failed

2. Inspect the pod:

bash
kubectl describe pod <pod-name>

Look for: -

State: Terminated

-

Reason: DeadlineExceeded

- events at the bottom showing deadline/termination 3. Check exact termination fields:

bash
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}{"\n"}{.status.containerStatuses[0].state.terminated.exitCode}{"\n"}{.status.phase}{"\n"}'

Interpretation: -

DeadlineExceeded

=> activeDeadlineSeconds - empty / null / something else => more likely worker restart / eviction / infra interruption How I’d interpret your two possibilities Case 1:

activeDeadlineSeconds

- strongest signal: Kubernetes explicitly says

DeadlineExceeded

- this is the cleanest confirmation Case 2: worker restarted multiple times - if the worker restart happened but the job pod itself shows

DeadlineExceeded

, then the log is most likely from the deadline timeout, not the worker restart - if there is no deadline signal and the pod vanished / was interrupted around the same timestamp as worker restarts, then the restart is the more likely cause Important nuance for Prefect 2.20.25 In Prefect 2.x, this message is generic for runtime cancellation. Prefect does not encode “this was specifically activeDeadlineSeconds” into that log line. So the reliable source of truth is: 1. Kubernetes job/pod reason 2. timestamps compared against worker restart times 3. worker logs around the same moment Recommended debugging workflow For one affected run, line up these timestamps: - task run crash time in Prefect - Kubernetes pod termination time - worker restart time Then: - if pod/job shows

DeadlineExceeded

at that same time → timeout caused it - if worker restarted at that same time and pod/job has no deadline reason → worker restart/infra event is more likely If you want, send me: -

kubectl describe pod <pod-name>

-

kubectl get job <job-name> -o yaml

- the relevant worker log snippet around the same timestamp and I can help you interpret which one it was.