My prefect worker stuck in Checking for cancelled ...
# ask-community
s
My prefect worker stuck in Checking for cancelled flow runs and never starts a flow run.
Copy code
04:10:54.775 | DEBUG   | prefect.runner - Checking for cancelled flow runs...
04:11:07.751 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-3bfdfc93-f00d-47a4-9d3c-765ad9a4bceb')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7fb79b197740>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7fb79d8c5490>>)
prefect version: 2.14.16 worker type: kubernetes (deployed with helm chart) I have go to prefect UI and deleted all canclling flow, still all flow run stuck in this stage
here is the log on UI (DEBUG is ruened on)
the logs of the pod
Copy code
(base) ➜  prefect git:(main) kubectl logs -f magnetic-beluga-fb4h4-9rhh7
03:59:53.795 | DEBUG   | prefect.profiles - Using profile 'default'
03:59:54.501 | DEBUG   | prefect.runner - Starting runner...
03:59:54.502 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:54.503 | DEBUG   | prefect.runner - Limit slot acquired for flow run '7491d933-a0b4-4927-8731-7caffa97eb52'
03:59:54.580 | INFO    | prefect.flow_runs.runner - Opening process...
03:59:54.582 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:54.585 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-3bfdfc93-f00d-47a4-9d3c-765ad9a4bceb')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7fb79b197740>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7fb79d8c5490>>)
03:59:54.586 | DEBUG   | prefect.runner - Checking for cancelled flow runs...
03:59:55.608 | DEBUG   | prefect.profiles - Using profile 'default'
<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
03:59:55.630 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:55.695 | INFO    | Flow run 'magnetic-beluga' - Downloading flow code from storage at '.'
03:59:55.696 | DEBUG   | Flow run 'magnetic-beluga' - Importing flow code from 'src/sat/day1/flow.py:day1_flow'
03:59:55.698 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:55.813 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:55.814 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:56.759 | DEBUG   | Flow run 'magnetic-beluga' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
03:59:56.760 | DEBUG   | prefect.task_runner.concurrent - Starting task runner...
03:59:56.777 | DEBUG   | prefect.engine - Received wait instruction for 3s: Scheduled time is in the future
03:59:59.891 | DEBUG   | Flow run 'day1_flow-20240305035956' - Renamed flow run 'magnetic-beluga' to 'day1_flow-20240305035956'
03:59:59.892 | DEBUG   | Flow run 'day1_flow-20240305035956' - Executing flow 'day1' for flow run 'day1_flow-20240305035956'...
03:59:59.893 | DEBUG   | Flow run 'day1_flow-20240305035956' - Beginning execution...
03:59:59.919 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
04:00:05.646 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-3bfdfc93-f00d-47a4-9d3c-765ad9a4bceb')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7fb79b197740>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7fb79d8c5490>>)
04:00:05.647 | DEBUG   | prefect.runner - Checking for cancelled flow runs...
nvm I realize this is normal log after tracing the code. The issue is actually due to our flow is unable to acquire concurrency context due to concurrency limit