Adam
07/29/2020, 3:12 PMjosh
07/29/2020, 3:18 PMaddress
to that Dask cluster to execute your tasks on its workers
• Using something like the DaskKubernetesEnvironment for dynamically spinning up a dask cluster for each run on k8sAdam
07/29/2020, 3:24 PMenvironment=DaskKubernetesEnvironment(min_workers=1, max_workers=3)
in my flow and then the Dask workers and scheduler will be auto provisioned on my cluster?josh
07/29/2020, 3:25 PMAdam
07/29/2020, 3:34 PMStorage
?josh
07/29/2020, 3:35 PMAdam
07/29/2020, 3:56 PMUnexpected error: RecursionError('maximum recursion depth exceeded')
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/engine/flow_runner.py", line 491, in get_flow_run_state
upstream_states = executor.wait(
File "/usr/local/lib/python3.8/site-packages/prefect/engine/executors/dask.py", line 375, in wait
return self.client.gather(futures)
File "/usr/local/lib/python3.8/site-packages/distributed/client.py", line 1982, in gather
return self.sync(
File "/usr/local/lib/python3.8/site-packages/distributed/client.py", line 832, in sync
return sync(
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 339, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 323, in f
result[0] = yield future
File "/usr/local/lib/python3.8/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/usr/local/lib/python3.8/site-packages/distributed/client.py", line 1876, in _gather
response = await future
File "/usr/local/lib/python3.8/site-packages/distributed/client.py", line 1927, in _gather_remote
response = await retry_operation(self.scheduler.gather, keys=keys)
File "/usr/local/lib/python3.8/site-packages/distributed/utils_comm.py", line 385, in retry_operation
return await retry(
File "/usr/local/lib/python3.8/site-packages/distributed/utils_comm.py", line 370, in retry
return await coro()
File "/usr/local/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
result = await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
response = await comm.read(deserializers=deserializers)
File "/usr/local/lib/python3.8/site-packages/distributed/comm/tcp.py", line 202, in read
msg = await from_frames(
File "/usr/local/lib/python3.8/site-packages/distributed/comm/utils.py", line 87, in from_frames
res = _from_frames()
File "/usr/local/lib/python3.8/site-packages/distributed/comm/utils.py", line 65, in _from_frames
return protocol.loads(
File "/usr/local/lib/python3.8/site-packages/distributed/protocol/core.py", line 130, in loads
value = _deserialize(head, fs, deserializers=deserializers)
File "/usr/local/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 302, in deserialize
return loads(header, frames)
File "/usr/local/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 64, in pickle_loads
return pickle.loads(x, buffers=buffers)
File "/usr/local/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 75, in loads
return pickle.loads(x)
File "/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 1264, in __getattr__
value = self._xxx_field_to_index.get(name)
File "/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 1264, in __getattr__
value = self._xxx_field_to_index.get(name)
File "/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 1264, in __getattr__
value = self._xxx_field_to_index.get(name)
[Previous line repeated 974 more times]
RecursionError: maximum recursion depth exceeded
josh
07/29/2020, 5:11 PMAdam
07/29/2020, 5:12 PM