schwannden kuo
03/05/2024, 4:12 AM04:10:54.775 | DEBUG | prefect.runner - Checking for cancelled flow runs...
04:11:07.751 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-3bfdfc93-f00d-47a4-9d3c-765ad9a4bceb')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7fb79b197740>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7fb79d8c5490>>)
prefect version: 2.14.16
worker type: kubernetes (deployed with helm chart)
I have go to prefect UI and deleted all canclling flow, still all flow run stuck in this stageschwannden kuo
03/05/2024, 4:20 AMschwannden kuo
03/05/2024, 4:22 AM(base) ➜ prefect git:(main) kubectl logs -f magnetic-beluga-fb4h4-9rhh7
03:59:53.795 | DEBUG | prefect.profiles - Using profile 'default'
03:59:54.501 | DEBUG | prefect.runner - Starting runner...
03:59:54.502 | DEBUG | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:54.503 | DEBUG | prefect.runner - Limit slot acquired for flow run '7491d933-a0b4-4927-8731-7caffa97eb52'
03:59:54.580 | INFO | prefect.flow_runs.runner - Opening process...
03:59:54.582 | DEBUG | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:54.585 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-3bfdfc93-f00d-47a4-9d3c-765ad9a4bceb')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7fb79b197740>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7fb79d8c5490>>)
03:59:54.586 | DEBUG | prefect.runner - Checking for cancelled flow runs...
03:59:55.608 | DEBUG | prefect.profiles - Using profile 'default'
<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
03:59:55.630 | DEBUG | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:55.695 | INFO | Flow run 'magnetic-beluga' - Downloading flow code from storage at '.'
03:59:55.696 | DEBUG | Flow run 'magnetic-beluga' - Importing flow code from 'src/sat/day1/flow.py:day1_flow'
03:59:55.698 | DEBUG | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:55.813 | DEBUG | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:55.814 | DEBUG | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
03:59:56.759 | DEBUG | Flow run 'magnetic-beluga' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
03:59:56.760 | DEBUG | prefect.task_runner.concurrent - Starting task runner...
03:59:56.777 | DEBUG | prefect.engine - Received wait instruction for 3s: Scheduled time is in the future
03:59:59.891 | DEBUG | Flow run 'day1_flow-20240305035956' - Renamed flow run 'magnetic-beluga' to 'day1_flow-20240305035956'
03:59:59.892 | DEBUG | Flow run 'day1_flow-20240305035956' - Executing flow 'day1' for flow run 'day1_flow-20240305035956'...
03:59:59.893 | DEBUG | Flow run 'day1_flow-20240305035956' - Beginning execution...
03:59:59.919 | DEBUG | prefect.client - Connecting to API at <http://prefect-server.deep-sat.svc:4200/api/>
04:00:05.646 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-3bfdfc93-f00d-47a4-9d3c-765ad9a4bceb')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7fb79b197740>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7fb79d8c5490>>)
04:00:05.647 | DEBUG | prefect.runner - Checking for cancelled flow runs...
schwannden kuo
03/05/2024, 5:13 AM