Josh
06/21/2022, 10:16 PMUnexpected error: KilledWorker('{{TaskRunName}}', <WorkerState '<tcp://127.0.0.1:44295>', name: 1, status: closed, memory: 0, processing: 72>)
Any idea what might be causing this?Kevin Kho
Josh
06/21/2022, 10:17 PMKevin Kho
Josh
06/21/2022, 10:21 PMKevin Kho
Josh
06/21/2022, 10:28 PMTraceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/flow_runner.py", line 543, in get_flow_run_state
{e: state for e, state in upstream_states.items()}
File "/usr/local/lib/python3.7/site-packages/prefect/executors/dask.py", line 448, in wait
return self.client.gather(futures)
File "/usr/local/lib/python3.7/site-packages/distributed/client.py", line 2152, in gather
asynchronous=asynchronous,
File "/usr/local/lib/python3.7/site-packages/distributed/utils.py", line 310, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "/usr/local/lib/python3.7/site-packages/distributed/utils.py", line 376, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.7/site-packages/distributed/utils.py", line 349, in f
result = yield future
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.7/site-packages/distributed/client.py", line 2009, in _gather
raise exception.with_traceback(traceback)
distributed.scheduler.KilledWorker: ('{{TaskRunName}}', <WorkerState '<tcp://127.0.0.1:44295>', name: 1, status: closed, memory: 0, processing: 72>)
Kevin Kho
Josh
06/21/2022, 10:29 PMKevin Kho
Josh
06/21/2022, 10:34 PMDaskExecutor(
cluster_kwargs= {"processes": True, "threads_per_worker": 1, "silence_logs": logging.DEBUG}
)
per your recommendationKevin Kho
Josh
06/21/2022, 11:05 PMKevin Kho
Josh
06/21/2022, 11:40 PM