https://prefect.io logo
Join Slack
Powered by
Hi <@ULVA73B9P> September 17, 2025, 11:30 06:00...
# ask-community
k
Hi @Marvin September 17, 2025, 11:30 060052.368 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-97f5d9b5-1991-48c7-9085-7f70bdbe634e')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7f53df5d3c10>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7f53dab61c40>>) prefect-ats-container September 17, 2025, 11:30 060052.368 | DEBUG | prefect.runner - Checking for cancelled flow runs... prefect-ats-container September 17, 2025, 11:30 060043.065 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-97f5d9b5-1991-48c7-9085-7f70bdbe634e')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7f53df5d3c10>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7f53dab61c40>>) prefect-ats-container September 17, 2025, 11:30 060043.066 | DEBUG | prefect.runner - Checking for cancelled flow runs... prefect-ats-container September 17, 2025, 11:30 060034.453 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-97f5d9b5-1991-48c7-9085-7f70bdbe634e')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7f53df5d3c10>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7f53dab61c40>>) prefect-ats-container September 17, 2025, 11:30 060034.454 | DEBUG | prefect.runner - Checking for cancelled flow runs... prefect-ats-container September 17, 2025, 11:30 060022.673 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-97f5d9b5-1991-48c7-9085-7f70bdbe634e')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7f53df5d3c10>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7f53dab61c40>>) prefect-ats-container September 17, 2025, 11:30 060022.674 | DEBUG | prefect.runner - Checking for cancelled flow runs... prefect-ats-container September 17, 2025, 11:30 060012.081 | DEBUG | prefect.runner - Checking for cancelled flow runs... prefect-ats-container this is an ecs task's logs, the same log has repetaed fro soemtime and teh conatimer exited, explain
m
✅ thought for 248.4 seconds
Thanks for the snippet — that DEBUG line is expected behavior from Prefect’s runner. What you’re seeing: - 
prefect.utilities.services.critical_service_loop - Starting run of functools.partial(... Runner._check_for_cancelled_flow_runs ...)
- 
prefect.runner - Checking for cancelled flow runs...
What it means: - Prefect 3 starts a background “cancellation watcher” while a flow run is executing. It’s implemented with 
critical_service_loop(...)
, which periodically calls 
Runner._check_for_cancelled_flow_runs
to see if the run was cancelled from the API. - At DEBUG level, it logs once per tick (typically every few seconds). So seeing these lines repeat is normal while the flow run is active or waiting. - Reference: critical_service_loop API Why the container exited: - In ECS, the container that runs the flow is ephemeral. When the flow run completes, fails, or is cancelled, the process exits and the ECS task stops. If you only see the repeating “checking for cancelled flow runs” and then the container exits, the most common explanations are: - The flow run finished (exit code 0). - The flow run was cancelled or errored (non-zero exit) and the task stopped. - ECS stopped the task (e.g., stop/termination/timeout/OOM) — Prefect’s cancellation loop logs can still appear up to the moment the process is terminated. How to confirm what happened: - Check the flow run’s state and logs in the Prefect UI around that timestamp to see if it reached Completed/Failed/Cancelled. - Check the ECS task’s Stop code/Reason and exit code in CloudWatch/ECS console. - Ensure you’re looking at INFO logs too — the cancellation loop is DEBUG; INFO-level lines (e.g., run start/end, state changes) will be more telling. You can set:
Copy code
PREFECT_LOGGING_LEVEL=INFO
- If you expected this container to be long-lived, note: the long-running component is the Prefect worker that polls a work pool; flow-run containers are short-lived by design. To run a persistent worker in ECS, you’d start it with something like:
Copy code
prefect worker start -p <your-work-pool-name>
(This is the correct command in Prefect 3 — verified with 
prefect worker start --help
.) A couple of clarifying questions to help pinpoint the cause: - Was this container the flow-run container (created per run) or your long-lived worker? - What was the ECS task’s exit code and Stop reason? - Do you see a terminal state for the corresponding flow run in the Prefect UI (Completed/Failed/Cancelled)? - Which Prefect version are you on, and what command/entrypoint did the ECS task run? Share those details and I can help trace the exact exit condition.
Open in Slack
PreviousNext
Powered by