https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have a zombie flow - in the UI it s...
# ask-marvin
o
@Marvin I have a zombie flow - in the UI it shows as running for 1.5 days, while the last step in the logs is Finished in state Completed() for task run (which aligns with the last task in the flow). This deployment has had zombie flows before as well, not a problem i have a whole lot else where. I have docker worker pools and docker workers that create containers to run each flow. Now for the UI I can of course simply delete this one to "get rid of it", but how can i make sure that it is not using resources on my machine? And how can I debug this deployment to get a guess as to why it at times creates these zombie flows?
m
✅ thought for 374.6 seconds
Thanks for the details — this does sound like a “zombie” flow: the engine stopped reporting to the API, so the flow run stayed Running even though your last task logged Completed. Here’s how to 1) make sure it’s not consuming resources and 2) track down root cause and harden against it. Immediate safety checks and cleanup - Verify/stop any leftover container for that flow run:
Copy code
# get infra pid (container id) from Prefect
  prefect flow-run inspect <FLOW_RUN_ID>

  # then check Docker
  docker ps -a | grep <CONTAINER_ID>
  docker logs --tail 200 <CONTAINER_ID>

  # if still running or stuck, stop and remove
  docker stop <CONTAINER_ID> && docker rm <CONTAINER_ID>
If the container is gone or stopped, it’s not using CPU; removing it ensures no disk/memory is tied up by an exited container. - Prefer Cancel over Delete in the UI/CLI - Cancel will signal the worker to kill infrastructure; Delete only removes the record. - CLI for reference:
Copy code
prefect flow-run cancel <FLOW_RUN_ID>
    prefect flow-run delete <FLOW_RUN_ID>
Hardening to prevent/recover from zombies - Enable runner heartbeats so Prefect can detect stalls and auto-recover - Set an env var in your Docker job to emit periodic heartbeats: - 
PREFECT_RUNNER_HEARTBEAT_FREQUENCY=60
(any integer ≥ 30 enables heartbeats) - Create an Automation to act when heartbeats are missed (two common choices): - Mark as Crashed when no heartbeat is seen within your window (e.g., ~2–3 minutes) - Or Cancel the run to ensure infrastructure is killed - UI: Automations -> New automation -> Proactive -> “When a flow run heartbeat is missing …” -> Action: Set state to Crashed (or Cancel) - Docs: Detect zombie flows, Event triggers - Ensure containers are cleaned up automatically - Set 
auto_remove: true
in your Docker worker job configuration so exited containers are removed by Docker: - In the Work Pool UI: edit your Docker pool’s base job template and set 
auto_remove
to 
true
- Or per-deployment with 
job_variables
- Example deployment (Prefect 3.x):
Copy code
from prefect import flow

    @flow
    def my_flow():
        ...

    if __name__ == "__main__":
        my_flow.from_source(
            source="<https://github.com/your-org/your-repo>",
            entrypoint="flows/my_flow.py:my_flow",
        ).deploy(
            name="my-docker-deployment",
            work_pool_name="your-docker-pool",
            job_variables={
                "image": "your-image:latest",
                "auto_remove": True,
                "env": {"PREFECT_RUNNER_HEARTBEAT_FREQUENCY": "60"},
            },
        )
- Docker worker config reference: 
auto_remove
is supported by the Docker worker job configuration. - Add a “long running” safety net - If you know how long a flow should run, set a flow-level timeout:
Copy code
@flow(timeout_seconds=3600)
    def my_flow():
        ...
- You can also add an Automation: “When a flow run has been Running for > X hours -> Cancel”. - Increase observability for the next occurrence - Set worker logs to DEBUG to capture more around submission/teardown:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
    prefect worker start -p <POOL_NAME>
- When you see a zombie, check: - Worker logs around the run’s start/finish - Container logs (
docker logs <CONTAINER_ID>
) - Whether the flow ever logged a flow-level completion (e.g., “Flow run finished in state Completed(...)”) — if your last log is a task completion, the engine likely crashed or lost connectivity after tasks completed but before finalizing the flow state. Likely causes to investigate - Container/engine exits or network blip after the last task finishes but before the engine posts the final flow state.
- Worker restart or crash mid-run (log streaming cuts out; container continues then exits silently). - Finalization blocked by non-daemon threads/subprocesses; container exits without a final state event reaching the API. - API connectivity issues from the container to the Prefect API near the run’s end. Related docs for deeper dives - Detecting and auto-handling zombie flows: Detect zombie flows - States overview: States - Worker healthchecks: Worker healthchecks A couple questions to help pinpoint the root cause: - Which Prefect version (server/agent image and Python package) and Docker worker version are you on? - Do you see a flow-level “Flow run finished in state Completed(…)” log for healthy runs, and is it missing on these zombies? - Any worker restarts or network interruptions around the time these runs finish? - Are you already setting 
auto_remove: true
on your Docker jobs? If you can share a flow run ID and your worker logs around the end of that run, I can help trace exactly where it stalled.
o
@Marvin here are some more detailes. it is the subflow that is still running (parent crashed) - and that is normal pattern: (brrr) PS C:\qfin\qfin> prefect flow-run inspect b1f749d8-3198-4d0b-add5-d5917412b85f FlowRun( id='b1f749d8-3198-4d0b-add5-d5917412b85f', created=DateTime(2025, 10, 24, 22, 22, 24, 516929, tzinfo=Timezone('UTC')), updated=DateTime(2025, 10, 24, 22, 22, 24, 603000, tzinfo=Timezone('UTC')), name='daring-earthworm', flow_id='a7b83293-6206-494f-83d1-195f26f826a9', state_id=UUID('019a1850-df51-7e4b-9d2e-3cc2d852d6cd'), flow_version='4b17d8c3c92c61fce5aee9bde00f7a66', parameters={'stocks': '<DataFrame>', 'iterations': 3, 'kwargs': {'db_type': 'duckdb'}}, context={}, empirical_policy=FlowRunPolicy(retries=0, retry_delay=0, pause_keys=set()), tags=[], labels={'prefect.flow.id': 'a7b83293-6206-494f-83d1-195f26f826a9'}, parent_task_run_id=UUID('a5129752-72cb-4867-acf0-27af04b034fc'), run_count=1, expected_start_time=DateTime(2025, 10, 24, 22, 22, 24, 516247, tzinfo=Timezone('UTC')), start_time=DateTime(2025, 10, 24, 22, 22, 24, 593456, tzinfo=Timezone('UTC')), estimated_run_time=datetime.timedelta(days=1, seconds=85099, microseconds=521710), estimated_start_time_delta=datetime.timedelta(microseconds=77209), state=State( id='019a1850-df51-7e4b-9d2e-3cc2d852d6cd', type=StateType.RUNNING, name='Running', timestamp=datetime.datetime(2025, 10, 24, 22, 22, 24, 593456, tzinfo=TzInfo(UTC)), state_details=StateDetails( flow_run_id=UUID('b1f749d8-3198-4d0b-add5-d5917412b85f'), task_run_id=UUID('a5129752-72cb-4867-acf0-27af04b034fc'), transition_id=UUID('381b8dea-0625-4149-839d-f4302b2eb0f6') ) ), job_variables={}, state_type=StateType.RUNNING, state_name='Running' ) (brrr) PS C:\qfin\qfin> (brrr) PS C:\qfin\qfin> prefect flow-run inspect 019a0c6b-817b-72a3-814e-4cac3cb08ae3 FlowRun( id='019a0c6b-817b-72a3-814e-4cac3cb08ae3', created=DateTime(2025, 10, 22, 14, 56, 3, 456375, tzinfo=Timezone('UTC')), updated=DateTime(2025, 10, 25, 11, 53, 23, 743000, tzinfo=Timezone('UTC')), name='kind-robin', flow_id='35cd6bd5-ec02-4da2-a26f-90e87efd30af', state_id=UUID('019a1b37-59bb-7037-995c-c6df6b13123e'), deployment_id=UUID('b8538213-ebc3-4486-987a-46d31ab462a6'), deployment_version='a4f3fff4', work_queue_name='default', flow_version='1abd1d9db0d3bbe7b7b7e6f7ec07aba8', parameters={}, idempotency_key='scheduled b8538213-ebc3-4486-987a-46d31ab462a6 03851a75-5216-4ca3-8926-b8b5036cd504 2025-10-24 222200+00:00', context={}, empirical_policy=FlowRunPolicy(retries=0, retry_delay=0, pause_keys=set()), tags=['auto-scheduled', 'production', 'nordnet_extract'], labels={ 'prefect.flow.id': '35cd6bd5-ec02-4da2-a26f-90e87efd30af', 'prefect.deployment.id': 'b8538213-ebc3-4486-987a-46d31ab462a6', 'prefect.worker.name': 'DockerWorker ccdc3b95-8b25-4d9c-a6a6-ac76bb14c12c', 'prefect.worker.type': 'docker', 'prefect.work-pool.name': 'production-pool', 'prefect.work-pool.id': 'c712cad6-6806-4769-ac87-bc55651be4db' }, run_count=1, expected_start_time=DateTime(2025, 10, 24, 22, 22, 0, tzinfo=Timezone('UTC')), start_time=DateTime(2025, 10, 24, 22, 22, 20, 395166, tzinfo=Timezone('UTC')), end_time=DateTime(2025, 10, 25, 11, 53, 23, 644248, tzinfo=Timezone('UTC')), total_run_time=datetime.timedelta(seconds=48663, microseconds=249082), estimated_run_time=datetime.timedelta(seconds=48663, microseconds=249082), estimated_start_time_delta=datetime.timedelta(seconds=20, microseconds=395166), auto_scheduled=True, infrastructure_pid='http+docker://localhost:0f3cd5bd56d434d7192965fab51d060c2e18b5b0f8d31d8a0f8fe0f9acbe9f18', created_by=CreatedBy(id=UUID('03851a75-5216-4ca3-8926-b8b5036cd504'), type='SCHEDULE', display_value='CronSchedule'), work_queue_id=UUID('09012307-9413-41d3-90c0-33005637bd37'), work_pool_id=UUID('c712cad6-6806-4769-ac87-bc55651be4db'), work_pool_name='production-pool', state=State( id='019a1b37-59bb-7037-995c-c6df6b13123e', type=StateType.CRASHED, name='Crashed', timestamp=datetime.datetime(2025, 10, 25, 11, 53, 23, 644248, tzinfo=TzInfo(UTC)), message='Flow run process exited with non-zero status code -9.', state_details=StateDetails( flow_run_id=UUID('019a0c6b-817b-72a3-814e-4cac3cb08ae3'), transition_id=UUID('dd7fc4f8-7fa3-42b9-a0dc-c0aae6ecd4f6') ) ), job_variables={}, state_type=StateType.CRASHED, state_name='Crashed' ) Not sure what here constitutes the docker container id. When i look in docker desktop, the only running containers are those that i expect, but not sure if a zombie flow container would show up there
m
✅ thought for 56.7 seconds
Super helpful detail — thanks. A few key takeaways from what you shared: - The “zombie” is a subflow run that’s still Running. - The parent flow run has already been marked Crashed with “Flow run process exited with non-zero status code -9.” - Subflows run in the same container/process as the parent. When the parent container is killed, subflows don’t get a final state unless the engine had a chance to report it. That’s why you see a stuck subflow but no corresponding Docker container. How to tell if it’s using resources - Subflows don’t have their own container, so they won’t show up separately in Docker Desktop. In your case the parent’s 
infrastructure_pid
shows the container id: - From your parent run:
Copy code
infrastructure_pid='<http+docker://localhost:0>f3cd5bd56d434d7192965fab51d060c2e18b5b0f8d31d8a0f8fe0f9acbe9f18'
The container id is the long suffix:
Copy code
f3cd5bd56d434d7192965fab51d060c2e18b5b0f8d31d8a0f8fe0f9acbe9f18
- Check its status/logs:
Copy code
docker ps -a --no-trunc --filter id=f3cd5bd56d434d7192965fab51d060c2e18b5b0f8d31d8a0f8fe0f9acbe9f18
  docker logs --tail 200 f3cd5bd56d434d7192965fab51d060c2e18b5b0f8d31d8a0f8fe0f9acbe9f18
  docker inspect -f '{{.State.Status}} Exit={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' f3cd5b...
- If the container is stopped (very likely, given the parent has an end_time and Crashed), it is not consuming CPU. You can safely remove it:
Copy code
docker rm f3cd5b...
Why this happens - Exit code -9 is a SIGKILL. In Docker this commonly indicates OOM kill or a hard kill (e.g., docker stop --time 0, host shutdown). - Because the process was killed abruptly, the Prefect engine did not get to set terminal states for in-flight subflows. Result: subflow stays Running in the API despite no container. Preventing and auto-recovering “zombie” subflows - Emit runner heartbeats and add an automation to crash/cancel stalled runs: - Set in your Docker job env:
Copy code
PREFECT_RUNNER_HEARTBEAT_FREQUENCY=60
(any integer ≥ 30 enables heartbeats) - In Automations (UI): create a proactive automation “when a flow-run heartbeat is missing within ~2–3 minutes” → Action: Set state to Crashed (or Cancel). This will catch both top-level and subflows when the process dies. - Docs: Detect zombie flows, Event triggers - Add a time-based guardrail as a backup - Automation: “When a flow run has been Running for > X hours → Set to Crashed (or Cancel)”. - This sweeps up old zombies even if heartbeats weren’t enabled yet. - Ensure containers clean up automatically - In your Docker work pool’s base job template (or per-deployment 
job_variables
), set:
Copy code
auto_remove: true
- This removes exited containers to avoid clutter. - Investigate/mitigate -9 kills (probable OOM) - Check if the container was OOMKilled:
Copy code
docker inspect -f '{{.State.OOMKilled}}' f3cd5b...
- If true, consider: - Set memory limits intentionally (so you know when it’s constrained) and adjust the workload:
Copy code
job_variables = {
        "mem_limit": "8g",
        "memswap_limit": "8g",  # or higher if you allow swap
      }
- Reduce concurrency inside the flow (e.g., thread/process pool size) or batch sizes. - Add retries/timeouts at the flow level:
Copy code
@flow(timeout_seconds=3600, retries=1, retry_delay_seconds=60)
      def my_flow(): ...
What to do with the current stuck subflow - It has no container attached, so it is not using resources. You can: - Cancel it:
Copy code
prefect flow-run cancel b1f749d8-3198-4d0b-add5-d5917412b85f
- Or delete it if you just want to clean the record:
Copy code
prefect flow-run delete b1f749d8-3198-4d0b-add5-d5917412b85f
Optional extra observability
- Run your worker with DEBUG to capture submission/teardown detail:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start -p production-pool
- Worker healthchecks: Worker healthchecks - States reference: States If you’d like, share: - Whether 
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
is currently set for this deployment - Whether 
auto_remove: true
is configured for your Docker jobs - The 
docker inspect
OOMKilled output for the parent container I can then suggest a concrete set of job_variables and an automation recipe tailored to your setup.
o
@Marvin i tried cancelling through the cli, but it seems stukc on cancelling now
m
✅ thought for 52.2 seconds
Got it — when a run gets stuck in Cancelling, it usually means Prefect sent a cancel signal but the underlying infrastructure is already gone, so the orchestration layer is waiting for a termination event that will never arrive. It’s not consuming resources at this point. Two ways to force-resolve it: Option A: Set state in the UI - Open the flow run page, use “Set state” (or “Mark as…”) and choose Cancelled or Crashed. - This is the quickest way to clear it when infra is gone. Option B: Force the state via the Python client - You can force a terminal state even if orchestration would normally reject it:
Copy code
import anyio
  from prefect.client.orchestration import get_client
  from prefect.states import Cancelled  # or Crashed

  async def force_cancel(flow_run_id: str):
      async with get_client() as client:
          await client.set_flow_run_state(flow_run_id, state=Cancelled(), force=True)

  anyio.run(force_cancel, "<FLOW_RUN_ID>")
- Replace 
Cancelled()
with 
Crashed()
if you prefer to reflect the abrupt termination. If you just want to remove the record:
Copy code
prefect flow-run delete <FLOW_RUN_ID>
Confirming resource usage - Your subflow shares the parent’s container. From your parent run: infrastructure_pid was:
Copy code
<http+docker://localhost:0>f3cd5bd56d434d7192965fab51d060c2e18b5b0f8d31d8a0f8fe0f9acbe9f18
The container ID is the long suffix. Check/cleanup:
Copy code
docker ps -a --no-trunc --filter id=f3cd5b...
  docker inspect -f '{{.State.Status}} Exit={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' f3cd5b...
  docker logs --tail 200 f3cd5b...
  # if stopped, it’s safe to remove
  docker rm f3cd5b...
- Since the parent is Crashed and has an end_time, that container is almost certainly stopped already and not using CPU. To prevent this next time - Enable runner heartbeats and an automation to mark missed-heartbeat runs Crashed/Cancelled (this will automatically clean up zombies, including subflows whose parent process died): - Add to your Docker job env:
Copy code
PREFECT_RUNNER_HEARTBEAT_FREQUENCY=60
- Automation in UI: proactive trigger “when a flow run heartbeat is missing for ~2–3 minutes” → Action: Set state to Crashed (or Cancel). - Docs: Detect zombie flows - Consider 
auto_remove: true
in your Docker job to auto-clean exited containers. If you share the flow run ID that’s stuck in Cancelling, I can give you a concrete 
force_cancel
one-liner you can run.
63 Views
Open in Slack
PreviousNext
Powered by