Short answer: “Cancelling” means Prefect received your cancel request and the worker is trying to stop the underlying infrastructure (process/container/pod). It will flip to “Cancelled” only after the worker confirms the infra is gone. If it sits in “Cancelling”, the worker likely didn’t pick up the event or couldn’t terminate the infra. How to check if it’s really stopped - UI - Open the flow run page and check: - Does it eventually show Cancelled (not Cancelling)? - Is an end time set? - Do logs contain messages like “Received cancellation” / “Killing infrastructure”? - CLI - Inspect details (state, end_time, infrastructure_pid):

prefect flow-run inspect <flow-run-id>

- View logs:

prefect flow-run logs <flow-run-id>

- List currently cancelling runs:

prefect flow-run ls --state CANCELLING

- Infra check (most definitive) - Subprocess/process: verify the PID from

infrastructure_pid

is no longer running - Docker:

docker ps

to confirm the container is gone - Kubernetes:

kubectl get pods/jobs

to confirm the job/pod is terminated Why runs can stick in “Cancelling” - The worker for that work pool is offline or not receiving cancellation events - Infra identifier mismatch or missing (e.g.,

infrastructure_pid

doesn’t match the machine/cluster actually running the work) - The infra refuses or fails to stop (e.g., app ignores SIGTERM; API error during kill) - Deployment was removed before cleanup finished How to resolve or force clean up 1) Ensure the right worker is online - The worker that started the run (same work pool/type) must be running to kill infra 2) Manually kill the infra (if it’s still running) - Process:

kill -9 <pid>

- Docker:

docker kill <container-id>

- Kubernetes:

kubectl delete job <job> -n <ns>

(or the pod) 3) Force-set the state to Cancelled (after infra is gone) - Python API:

from prefect.client.orchestration import get_client
  from prefect.client.schemas.objects import State, StateType

  async def force_cancel(flow_run_id: str):
      async with get_client() as client:
          await client.set_flow_run_state(
              flow_run_id=flow_run_id,
              state=State(type=StateType.CANCELLED, message="Force-cancelled"),
              force=True
          )

- Or, as a last resort once you’re sure nothing is running, delete the record:

prefect flow-run delete <flow-run-id>

Tips - Check logs for “Killing infrastructure” from the worker; if absent, the worker may be offline - If you see an

infrastructure_pid

but the process/container/pod is already gone, a quick force-cancel (above) will clear the stuck state If you can share: - The worker type (subprocess, Docker, Kubernetes) - How you triggered cancel (UI/CLI) - One affected flow-run ID (or a sample log snippet) …I can help pinpoint exactly why yours is lingering and the safest next step.

Great question — here’s where to run each type of command. Prefect CLI / API (inspect, logs, cancel, delete) - Where to run: Anywhere that can reach your Prefect API (your laptop, a jump host, CI, server, or worker). Not tied to “server-side” or “worker-side.” - Make sure the environment has

PREFECT_API_URL

(and

PREFECT_API_KEY

for Cloud) set. - Examples:

prefect flow-run inspect <flow-run-id>
  prefect flow-run logs <flow-run-id>
  prefect flow-run cancel <flow-run-id>
  prefect flow-run ls --state CANCELLING

- Force-cancel via Python (also anywhere with API access):

from prefect.client.orchestration import get_client
  from prefect.client.schemas.objects import State, StateType

  async def force_cancel(flow_run_id: str):
      async with get_client() as client:
          await client.set_flow_run_state(
              flow_run_id=flow_run_id,
              state=State(type=StateType.CANCELLED, message="Force-cancelled"),
              force=True
          )

Infrastructure-level commands (actually stopping what’s running) - Subprocess/Process worker - Where: On the same machine that launched the process (the worker host). - Example:

kill -9 <pid>

- Docker worker - Where: On the host whose Docker daemon is running the container (typically the worker host). If you use a remote Docker API (

DOCKER_HOST

), you can run it from anywhere that can reach that daemon. - Example:

docker kill <container-id>

- Kubernetes worker - Where: Anywhere that has

kubectl

access/permissions to the target cluster/namespace (not tied to worker or server machines). - Examples:

kubectl delete job <job-name> -n <namespace>
    kubectl delete pod <pod-name> -n <namespace>

Environment variables (for cancellation behavior) - Worker-side: -

PREFECT_WORKER_ENABLE_CANCELLATION=true

-

PREFECT_WORKER_CANCELLATION_POLL_SECONDS=10

(optional fallback polling) - Set these in the environment where the worker process runs (container, VM, etc.). - Server-side (self-hosted Prefect Server only): -

PREFECT_SERVER_SERVICES__CANCELLATION_CLEANUP__ENABLED=true

- This is configured on your Prefect Server service, not on workers. For Prefect Cloud, this service is managed by Prefect. UI checks - Where: Anywhere in a browser. Open the flow run page in the UI to confirm state transitions and view logs. Rule of thumb - If it’s a Prefect CLI/API action (cancel, inspect, logs, delete): run it anywhere with API access. - If it manipulates or verifies the actual compute (PID, container, pod): run it where that compute can be controlled (worker host for processes/containers; any admin machine with kube access for Kubernetes). If you tell me which worker type you’re using (subprocess, Docker, or Kubernetes) and where it’s running, I can point you to the exact machine/cluster where each step should happen.