Deepanshu Aggarwal
11/04/2022, 6:41 AMAttributeError: 'coroutine' object has no attribute 'type'
and the flow run crashes. any idea why this happened.
adding detailed logs in the commentEncountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 580, in orchestrate_flow_run
result = await run_sync(flow_call)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(call, cancellable=True)
File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "flows/calculate_usages_flows/usages_flow.py", line 44, in usages_flow
usages_initializer(scheduler_output['body_array'])
File "/usr/local/lib/python3.9/site-packages/prefect/flows.py", line 439, in __call__
return enter_flow_run_engine_from_flow_call(
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 154, in enter_flow_run_engine_from_flow_call
return run_async_from_worker_thread(begin_run)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 148, in run_async_from_worker_thread
return anyio.from_thread.run(call)
File "/usr/local/lib/python3.9/site-packages/anyio/from_thread.py", line 49, in run
return asynclib.run_async_from_thread(func, *args)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
return f.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 446, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 475, in create_and_begin_subflow_run
terminal_state = await orchestrate_flow_run(
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 627, in orchestrate_flow_run
await wait_for_task_runs_and_report_crashes(
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1254, in wait_for_task_runs_and_report_crashes
if not state.type == StateType.CRASHED:
AttributeError: 'coroutine' object has no attribute 'type'
Khuyen Tran
11/04/2022, 3:33 PMJeff Hale
11/04/2022, 9:17 PMJames Brady
11/14/2022, 4:59 PMdask_kubernetes.KubeCluster
15:31:27.888 | INFO | Flow run 'cornflower-manul' - Created task run 'download_reshape_upload-8c37f7f7-55' for task 'download_reshape_upload'
15:31:27.894 | INFO | Flow run 'cornflower-manul' - Submitted task run 'download_reshape_upload-8c37f7f7-55' for execution.
2022-11-14 15:43:15,039 - distributed.deploy.adaptive - INFO - Retiring workers [1]
2022-11-14 15:43:32,041 - distributed.deploy.adaptive - INFO - Retiring workers [7]
2022-11-14 15:43:58,039 - distributed.deploy.adaptive - INFO - Retiring workers [0]
2022-11-14 15:44:11,039 - distributed.deploy.adaptive - INFO - Retiring workers [8, 10, 11]
2022-11-14 15:44:48,040 - distributed.deploy.adaptive - INFO - Retiring workers [6]
2022-11-14 15:44:57,039 - distributed.deploy.adaptive - INFO - Retiring workers [3]
2022-11-14 15:45:19,039 - distributed.deploy.adaptive - INFO - Retiring workers [4, 5]
2022-11-14 15:45:21,039 - distributed.deploy.adaptive - INFO - Retiring workers [11]
15:45:27.270 | ERROR | Flow run 'cornflower-manul' - Encountered exception during execution:
Traceback (most recent call last):
File "/code/.venv/lib/python3.10/site-packages/prefect/engine.py", line 612, in orchestrate_flow_run
waited_for_task_runs = await wait_for_task_runs_and_report_crashes(
File "/code/.venv/lib/python3.10/site-packages/prefect/engine.py", line 1284, in wait_for_task_runs_and_report_crashes
if not state.type == StateType.CRASHED:
AttributeError: 'coroutine' object has no attribute 'type'
2022-11-14 15:45:27,277 - distributed.deploy.adaptive_core - INFO - Adaptive stop
2022-11-14 15:45:27,410 - distributed.deploy.adaptive_core - INFO - Adaptive stop
15:45:27.411 | ERROR | Flow run 'cornflower-manul' - Crash detected! Execution was interrupted by an unexpected exception: AttributeError: 'coroutine' object has no attribute 'type'
15:45:27.485 | ERROR | prefect.engine - Engine execution of flow run '1c4aac07-e116-47fe-a8e7-31de3cb613f8' exited with unexpected exception
Traceback (most recent call last):
File "/code/.venv/lib/python3.10/site-packages/prefect/engine.py", line 1634, in <module>
enter_flow_run_engine_from_subprocess(flow_run_id)
File "/code/.venv/lib/python3.10/site-packages/prefect/engine.py", line 181, in enter_flow_run_engine_from_subprocess
return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
File "/code/.venv/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/code/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/code/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/code/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/code/.venv/lib/python3.10/site-packages/prefect/engine.py", line 299, in retrieve_flow_then_begin_flow_run
return await begin_flow_run(
File "/code/.venv/lib/python3.10/site-packages/prefect/engine.py", line 361, in begin_flow_run
terminal_state = await orchestrate_flow_run(
File "/code/.venv/lib/python3.10/site-packages/prefect/engine.py", line 657, in orchestrate_flow_run
await wait_for_task_runs_and_report_crashes(
File "/code/.venv/lib/python3.10/site-packages/prefect/engine.py", line 1284, in wait_for_task_runs_and_report_crashes
if not state.type == StateType.CRASHED:
AttributeError: 'coroutine' object has no attribute 'type'
15:31:15.890 | INFO | prefect.task_runner.dask - Creating a new Dask cluster with `dask_kubernetes.classic.kubecluster.KubeCluster`
Creating scheduler pod on cluster. This may take some time.
2022-11-14 15:31:21,037 - distributed.deploy.adaptive - INFO - Adaptive scaling started: minimum=0 maximum=10
/code/.venv/lib/python3.10/site-packages/distributed/client.py:1348: VersionMismatchWarning: Mismatched versions found
+---------+----------------+----------------+---------+
| Package | Client | Scheduler | Workers |
+---------+----------------+----------------+---------+
| lz4 | None | 4.0.2 | None |
| python | 3.10.7.final.0 | 3.8.13.final.0 | None |
+---------+----------------+----------------+---------+
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
Zanie
11/15/2022, 7:39 PMJames Brady
11/16/2022, 3:11 PMDaskTaskRunner
and dask_kubernetes.KubeCluster
?
If not, is there a different infrastructure setup that we should consider?Deepanshu Aggarwal
11/16/2022, 3:35 PMJames Brady
11/16/2022, 3:37 PMDeepanshu Aggarwal
11/16/2022, 3:38 PMJames Brady
11/16/2022, 3:40 PMDeepanshu Aggarwal
11/16/2022, 3:44 PMAnna Geller
11/16/2022, 4:17 PMJames Brady
11/16/2022, 4:22 PMAnna Geller
11/16/2022, 4:25 PMalex
11/16/2022, 4:38 PMDaskTaskRunner
that works with a coiled cluster: https://github.com/PrefectHQ/prefect-dask/issues/26#issuecomment-1258755216. This portion of the prefect-dask
documentation may also help with distributing work across workers: https://prefecthq.github.io/prefect-dask/#distributing-dask-collections-across-workersJeff Hale
11/17/2022, 2:04 AMMichiel Verburg
11/18/2022, 3:04 PMJames Brady
11/18/2022, 3:07 PMZanie
11/18/2022, 3:08 PMDeepanshu Aggarwal
11/18/2022, 3:08 PMMichiel Verburg
11/18/2022, 3:09 PMDeepanshu Aggarwal
11/18/2022, 3:10 PMMichiel Verburg
11/18/2022, 3:20 PMRyan Peden
11/18/2022, 3:37 PMOscar Björhn
11/18/2022, 3:57 PMDeepanshu Aggarwal
11/18/2022, 4:07 PMRyan Peden
11/18/2022, 4:07 PMOscar Björhn
11/18/2022, 4:08 PMMichiel Verburg
11/21/2022, 9:17 AMOscar Björhn
11/21/2022, 9:38 AMAnna Geller
11/21/2022, 2:06 PMJon
02/01/2023, 10:58 PM