<@ULVA73B9P> we have a run in a cancelling state f...
# ask-marvin
l
@Marvin we have a run in a cancelling state for 15 hours - is there a way to force this to cancel?
m
thought for 228.5 seconds
Yes—there are a few ways to force it, depending on how hard you want to push. Assuming Prefect 3.x: Quick try first - CLI: attempt a normal cancel
Copy code
prefect flow-run cancel <FLOW_RUN_ID>
Force the state to Cancelled (API/Client) - This bypasses the worker and flips Prefect’s record to Cancelled. It will not kill the underlying job; you’ll need to stop it in your infra (Kubernetes job/pod, ECS task, Docker container, local process). Async client:
Copy code
from prefect.client.orchestration import get_client
from prefect.states import Cancelled
import asyncio

async def force_cancel(flow_run_id: str):
    async with get_client() as client:
        await client.set_flow_run_state(flow_run_id, state=Cancelled(), force=True)

asyncio.run(force_cancel("<FLOW_RUN_ID>"))
Sync client:
Copy code
from prefect.client.orchestration import SyncPrefectClient
from prefect.states import Cancelled

with SyncPrefectClient() as client:
    client.set_flow_run_state("<FLOW_RUN_ID>", state=Cancelled(), force=True)
API reference: Set flow run state API Notes - “Cancelling” for hours usually means the worker never acknowledged the cancel (worker offline/crashed) or the infrastructure job is stuck. After forcing the state above, make sure to terminate the actual job in your environment. - If you just want to remove the record after forcing the state, you can delete it:
Copy code
prefect flow-run delete <FLOW_RUN_ID>
If you need help stopping the underlying job, let me know your worker/infrastructure (Kubernetes, ECS, Docker, local) and I can share the exact steps. Also, do you use Prefect Cloud or self-hosted UI?
l
we use a self hosted ui
@Marvin we use a self hosted ui
m
thought for 53.5 seconds
Got it — for a self‑hosted UI: - The UI’s Cancel button just requests cancellation. If the worker never acknowledges (offline/crashed), the run can sit in “Cancelling”. There isn’t a “force cancel” button in the UI today. Two reliable ways to force it: 1) CLI (normal cancel) - Works if the server will accept the transition:
Copy code
prefect flow-run cancel <FLOW_RUN_ID>
2) Force the state via API/Client (bypasses the worker) - This flips Prefect’s record to Cancelled. It will not kill the underlying job — you’ll need to stop it on your infra (K8s/ECS/Docker/etc.). Python client:
Copy code
from prefect.client.orchestration import SyncPrefectClient
from prefect.states import Cancelled

with SyncPrefectClient() as client:
    client.set_flow_run_state("<FLOW_RUN_ID>", state=Cancelled(), force=True)
cURL (direct REST call to your server):
Copy code
curl -X POST \
  "<YOUR_PREFECT_API_URL>/flow_runs/<FLOW_RUN_ID>/set_state" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <API_TOKEN_IF_CONFIGURED>" \
  -d '{
        "state": {
          "type": "CANCELLED",
          "name": "Cancelled",
          "message": "Force cancel via API"
        },
        "force": true
      }'
API reference: Set flow run state API After forcing the state, be sure to terminate the actual workload: - Kubernetes:
kubectl delete job <job>
or
kubectl delete pod <pod>
- Docker:
docker stop <container>
(then
docker rm
) - ECS: stop the task in the service/cluster - Process/local: kill the PID on the worker host If you tell me which work pool/infrastructure you’re using (Kubernetes, Docker, ECS, Process, etc.), I can provide exact commands to safely stop the underlying job.
l
@Marvin kubernetes
m
thought for 75.5 seconds
Great — here’s how to cleanly force-cancel and stop the underlying Kubernetes workload. Step 1 — Force cancel the Prefect flow run - This flips Prefect’s state to Cancelled so the worker won’t try to keep/respawn it.
Copy code
from prefect.client.orchestration import SyncPrefectClient
from prefect.states import Cancelled

with SyncPrefectClient() as client:
    client.set_flow_run_state("<FLOW_RUN_ID>", state=Cancelled(), force=True)
Or via REST:
Copy code
curl -X POST \
  "<YOUR_PREFECT_API_URL>/flow_runs/<FLOW_RUN_ID>/set_state" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <API_TOKEN_IF_CONFIGURED>" \
  -d '{
        "state": {"type": "CANCELLED", "name": "Cancelled", "message": "Force cancel via API"},
        "force": true
      }'
API ref: Set flow run state API Step 2 — Delete the Kubernetes Job/Pod for that run - Prefect labels k8s resources with the flow run ID; commonly:
<http://prefect.io/flow-run-id=<FLOW_RUN_ID|prefect.io/flow-run-id=<FLOW_RUN_ID>>
. If you customized the job template, adjust the selector accordingly. Find the resources:
Copy code
FLOW_RUN_ID=<FLOW_RUN_ID>
kubectl get job,pod -A -l <http://prefect.io/flow-run-id=$FLOW_RUN_ID|prefect.io/flow-run-id=$FLOW_RUN_ID> -o wide
Delete the Job (preferred; it will cascade to Pods):
Copy code
kubectl delete job -n <NAMESPACE> -l <http://prefect.io/flow-run-id=$FLOW_RUN_ID|prefect.io/flow-run-id=$FLOW_RUN_ID> --cascade=foreground
If there’s only a Pod (or it’s stuck), force delete the Pod:
Copy code
kubectl delete pod -n <NAMESPACE> -l <http://prefect.io/flow-run-id=$FLOW_RUN_ID|prefect.io/flow-run-id=$FLOW_RUN_ID> --force --grace-period=0
Notes - Do Step 1 first so the worker doesn’t recreate the Job. - If you’re unsure of the namespace, use
-A
to locate it, then run the delete in that namespace. - If finalizers or termination hang, the
--force --grace-period=0
on the Pod command will hard-kill it. Kubernetes worker docs (config, templates, labels): Kubernetes deployment and workers If you paste the flow run ID here, I can give you the exact kubectl commands with your namespace.