Tim Galvin
11/24/2022, 12:59 PMEncountered exception during execution:
Traceback (most recent call last):
File "/software/projects/askaprt/tgalvin/setonix/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/prefect/engine.py", line 612, in orchestrate_flow_run
waited_for_task_runs = await wait_for_task_runs_and_report_crashes(
File "/software/projects/askaprt/tgalvin/setonix/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/prefect/engine.py", line 1317, in wait_for_task_runs_and_report_crashes
if not state.type == StateType.CRASHED:
AttributeError: 'coroutine' object has no attribute 'type'
I am running a known version of my workflow on a known dataset, which has worked perfectly fine dozens of times before. It seems to be saying the the state
above is not an orion
model -- rather a coroutine. All my tasks are using the normal task
decorator around normal non-async python functions.PREFECT_API_URL
variable to point to my self-hosted orion server. Could I / should I also be setting anything related to URL or DB timeouts as well do you think?Ryan Peden
11/24/2022, 7:01 PMTim Galvin
11/25/2022, 7:39 AMdask_jobqeue.SLURMCluster
to fire up the dask-workers
on the slurm resource.
I think I made a connection -- it seems like one of the Dask workers in my set are being killed iwth a Sig bus
error. I am running this on a fairly new cluster, so it might just be some teething issues coming from that.Ryan Peden
11/26/2022, 1:15 AMTim Galvin
11/26/2022, 3:14 AMSig Bus
error outside of the python dask-worker, killing the python code outright entirely. I am trying to run this code on a new HPC cluster that is still experiencing setup issues, it thinkRyan Peden
11/26/2022, 4:01 AM'coroutine' object has no attribute 'type'
when something goes wrong in a Dask worker.Tim Galvin
11/26/2022, 6:15 AMRyan Peden
12/02/2022, 12:59 AM