Short answer: deleting the flow run from the UI will remove it from Prefect, but it will NOT stop whatever is actually running (process/container/pod). So the UI will be “clean,” but the underlying job may keep running and consuming resources. Why it’s stuck in Cancelling - Cancelling is a transient state while a worker tries to stop the underlying infrastructure. If it’s stuck for days, common causes are: - The worker that started the run is down or not connected, so it never processes the cancellation - The run is missing an

infrastructure_pid

, so the worker doesn’t know what to kill - The container/pod/process was killed outside Prefect, and the worker never reconciled the state What I recommend 1) Restart the worker for the work pool that ran this flow. Workers reconcile “Cancelling” runs and will usually finish the job once they’re online. 2) Inspect the run to see the infrastructure identifiers:

prefect flow-run inspect <flow-run-id>

Look for

infrastructure_pid

to learn what to kill: - Subprocess: hostname:PID → kill the PID on that host - Docker: docker-sock:container-id →

docker rm -f <container-id>

- Kubernetes: cluster:job/pod name →

kubectl delete job|pod ...

3) If you just want it gone from the UI and you’ve already stopped the underlying job, you can delete it:

prefect flow-run delete <flow-run-id>

Note: this removes the run and its logs from Prefect, but does not stop execution if it’s still running. Optional: retry cancel or force-finalize

prefect flow-run cancel <flow-run-id>

If that doesn’t move, you can force-set the state to Cancelled via the API; happy to share a short snippet if you want that route. Helpful references - Cancellation concepts: states.py - Flow-run deletion server code: server/models/flow_runs.py If you can share which work pool/infrastructure you used (subprocess, Docker, Kubernetes) and whether your worker is currently running, I can give you exact commands to clean it up safely.

Yes. In Prefect 3.x,

infrastructure_pid

is a field on the FlowRun object and you can fetch it via both the Python client and the REST API. Python client (async and sync):

# Async
from uuid import UUID
from prefect.client.orchestration import PrefectClient

async with PrefectClient() as client:
    run = await client.read_flow_run(UUID("<flow-run-id>"))
    print(run.infrastructure_pid)  # may be None if not recorded yet

# Sync
from uuid import UUID
from prefect.client.orchestration import SyncPrefectClient

with SyncPrefectClient() as client:
    run = client.read_flow_run(UUID("<flow-run-id>"))
    print(run.infrastructure_pid)

REST API:

# Single flow run
curl -s -H "Authorization: Bearer <API_KEY>" \
  <https://<your-prefect-api>/api/flow_runs/<flow-run-id>>

# Response includes:
# {
#   "id": "...",
#   "name": "...",
#   "infrastructure_pid": "host:12345" | "unix:///var/run/docker.sock:abc123" | null,
#   ...
# }

# Filter/query multiple flow runs
curl -s -H "Authorization: Bearer <API_KEY>" -H "Content-Type: application/json" \
  -X POST <https://<your-prefect-api>/api/flow_runs/filter> \
  -d '{
        "flow_runs": {"state": {"type": ["RUNNING"]}},
        "limit": 10
      }'
# Each item in the response array includes "infrastructure_pid"

Notes: -

infrastructure_pid

can be null if the run hasn’t started yet or the worker didn’t record it. - Task runs do not have an `infrastructure_pid`; it’s only on flow runs. References: - server/api/flow_runs.py - server/schemas/responses.py

Got it — that makes sense. For deployments running on static infrastructure (e.g., started via flow.serve/serve), it’s normal for

infrastructure_pid

to be None. There’s no separate worker reporting a PID back to the API; the PID is tracked only inside the long-running serve process. If that serve process is offline or lost track of the child subprocess, Prefect can’t cancel it and the run may sit in Cancelling. What you can do next - If the serve process is still running on the host: - Find and kill the child flow subprocess from that host. On Linux, you can identify it by the environment variable `PREFECT__FLOW_RUN_ID=<your-run-id>`:

# List PIDs whose environment contains the flow run id
  sudo bash -c 'for p in /proc/[0-9]*; do tr "\0" "\n" < $p/environ 2>/dev/null | grep -q "PREFECT__FLOW_RUN_ID=<flow-run-id>" && echo ${p##*/}; done'

  # Then, force kill the PID(s)
  sudo kill -9 <pid>

- Alternatively, if you know the serve process PID, inspect its children:

# Replace <serve-pid> with the PID of the serve process
  ps --ppid <serve-pid> -o pid,ppid,cmd

- If the serve process is gone: - You’ll need to find any orphaned subprocess on that host (the /proc scan above still works on Linux). - If you can’t access the host, you can force-finalize the run in Prefect (this does not stop the process):

prefect flow-run cancel <flow-run-id>
    # or set to Cancelled programmatically with force=True; I can share that snippet if helpful

Reminder - Deleting the flow run in the UI or via CLI cleans up Prefect’s records and logs, but will not stop the underlying process. - If you were actually using a process work pool + worker (vs. serve),

infrastructure_pid

should usually be populated. If it isn’t, the worker likely failed to set it due to a startup or API update error; the worker logs will show that. References - Process worker (PID reporting and submission): workers/process.py - Runner startup handshake (

task_status.started(pid)

): runner/runner.py - Flow run environment variables (

PREFECT__FLOW_RUN_ID

): workers/base.py If you can share whether this was launched via

flow.serve()

and which host it ran on, I can tailor exact kill/find commands for your OS.

Absolutely. Here are user-space commands you can run without sudo to find and stop the flow subprocess by its PREFECT__FLOW_RUN_ID. Replace <flow-run-id> with your actual run ID. Linux (procfs available) - Find the PID(s) of your run:

FLOW_RUN_ID=<flow-run-id>
for f in /proc/[0-9]*/environ; do
  [ -r "$f" ] || continue
  tr '\0' '\n' < "$f" 2>/dev/null | grep -q "PREFECT__FLOW_RUN_ID=$FLOW_RUN_ID" && \
    echo "${f%/environ}" | awk -F'/' '{print $3}'
done

- Optionally confirm the process before killing:

PID=<pid-from-above>
ps -fp "$PID"

- Terminate the process group (kills the flow and its children):

PID=<pid-from-above>
PGID=$(ps -o pgid= -p "$PID" | tr -d ' ')
kill -TERM "-$PGID"
sleep 5
kill -KILL "-$PGID"  # if still running

macOS - Find the PID(s) for your user:

FLOW_RUN_ID=<flow-run-id>
for pid in $(ps -x -o pid=); do
  ps ewwp "$pid" 2>/dev/null | grep -q "PREFECT__FLOW_RUN_ID=$FLOW_RUN_ID" && echo "$pid"
done

- Confirm and terminate (same as Linux):

PID=<pid-from-above>
ps -fp "$PID"

PGID=$(ps -o pgid= -p "$PID" | tr -d ' ')
kill -TERM "-$PGID"
sleep 5
kill -KILL "-$PGID"

Notes - Permission denied messages are normal; you’ll only be able to read env for processes you own. - Sending the signal to the negative PGID ensures any child processes spawned by the flow are also terminated. - The environment variable is set by Prefect for each flow subprocess:

PREFECT__FLOW_RUN_ID=<run-id>

(see workers/base.py). If you run into issues on your OS/shell, tell me which platform you’re on and I’ll tailor the one-liner.

Yes—if your search didn’t find any PIDs with PREFECT__FLOW_RUN_ID for your user, it’s very likely the process has already exited. In that case, the run lingering in Cancelling is usually because the static serve process wasn’t around to finish the state transition, not because you cancelled the child first. Cancelling a child flow first is supported; it shouldn’t by itself cause the parent to be stuck in Cancelling. What I recommend now - Quick double-check there’s nothing left for your user: - Linux/macOS:

ps -u "$USER" -o pid,command | grep -E 'prefect(\.engine| engine)|python .*prefect.*engine' | grep -v grep

If that’s empty, you’re safe to finalize in Prefect. - Force-finalize the parent run to Cancelled (this won’t try to kill anything, it just cleans up the state in Prefect):

from uuid import UUID
from prefect.client.orchestration import PrefectClient
from prefect.states import State, StateType

async def force_cancel(flow_run_id: str):
    async with PrefectClient() as client:
        await client.set_flow_run_state(
            flow_run_id=UUID(flow_run_id),
            state=State(type=StateType.CANCELLED, message="Force-cancelled after manual check"),
            force=True,
        )

# await force_cancel("<flow-run-id>")

- Alternatively, if you don’t need the record anymore, you can delete it:

prefect flow-run delete <flow-run-id>

If you want, share the OS you’re on and I can tailor a one-liner to double-check for any stray processes for your user without sudo.

Thanks, that helps. Since the flow subprocess ran inside a Debian container on a Red Hat host, two things can make your earlier host-side search return nothing: - The subprocess already exited. - The subprocess is still running but inside the container as a different user (commonly root), so you can’t read /proc/<pid>/environ on the host without sudo. If you have access to the container runtime, here’s how to search and clean up from inside the Debian container without needing host sudo. Docker - Find the container name/ID for your serve container (the long-running process where the flow was launched).

docker ps --format 'table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Status}}'

- Exec into it and search for the flow subprocess by environment var, then kill its process group:

# Replace <container> and <flow-run-id>
docker exec -it <container> bash -lc '
FLOW_RUN_ID=<flow-run-id>
PIDS=$(for f in /proc/[0-9]*/environ; do
  [ -r "$f" ] || continue
  tr "\0" "\n" < "$f" 2>/dev/null | grep -q "PREFECT__FLOW_RUN_ID=$FLOW_RUN_ID" && \
    echo "${f%/environ}" | awk -F/ "{print \$3}"
done)
echo "Found PIDs: $PIDS"
for PID in $PIDS; do
  PGID=$(ps -o pgid= -p "$PID" | tr -d " ")
  echo "Terminating PGID $PGID"
  kill -TERM "-$PGID" || true
done
sleep 5
for PID in $PIDS; do
  PGID=$(ps -o pgid= -p "$PID" | tr -d " ")
  kill -KILL "-$PGID" || true
done
'

Podman (common on RHEL) - Same idea with podman:

podman ps --format 'table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Status}}'

podman exec -it <container> bash -lc '
FLOW_RUN_ID=<flow-run-id>
PIDS=$(for f in /proc/[0-9]*/environ; do
  [ -r "$f" ] || continue
  tr "\0" "\n" < "$f" 2>/dev/null | grep -q "PREFECT__FLOW_RUN_ID=$FLOW_RUN_ID" && \
    echo "${f%/environ}" | awk -F/ "{print \$3}"
done)
echo "Found PIDs: $PIDS"
for PID in $PIDS; do
  PGID=$(ps -o pgid= -p "$PID" | tr -d " ")
  echo "Terminating PGID $PGID"
  kill -TERM "-$PGID" || true
done
sleep 5
for PID in $PIDS; do
  PGID=$(ps -o pgid= -p "$PID" | tr -d " ")
  kill -KILL "-$PGID" || true
done
'

If those commands find no PIDs inside the container, the subprocess has almost certainly exited already. Why Cancelling can linger here - With static infrastructure, the long-running serve container is responsible for reconciling cancellations. If it’s stopped or lost track of the child process, Prefect won’t transition the run to Cancelled automatically even if the child already died. Cancelling a child first should not by itself cause the parent to stick; the usual culprit is the serve process not being around to finish the transition. Next steps - If the subprocess is gone, you can safely force-finalize or delete the run in Prefect:

prefect flow-run cancel <flow-run-id>  # retry cancel
# or programmatically set to Cancelled with force=True (I can share a snippet)
# or delete it:
prefect flow-run delete <flow-run-id>

- If the serve container is still up, consider restarting it so it reconnects to Prefect and reconciles states. If you can confirm whether you’re using docker or podman and the container name, I can give you exact commands tailored to your setup.