Rob Fowler
09/15/2020, 10:28 AMNuno Silva
09/15/2020, 11:03 AMexecutor = DaskExecutor(
address=cluster.scheduler_address,
client_kwargs={"timeout": 20},
)
but I'll try the following then:
executor = DaskExecutor(
cluster_kwargs={"address": "<tcp://51.105.126.161:8786>", "timeout": 20},
)
"Cannot specify `address` and `cluster_class`/`cluster_kwargs`"
tiemout: Timeout duration for initial connection to the scheduler
I believe the problem is during normal call to the cluster to execute operations, hence I was trying to change the value: dask.config.set({"distributed.comm.timeouts.connect": 20})
. don't know how to pass this as an argument to client thoughJim Crist-Harif
09/15/2020, 1:42 PMClient
on a Worker
, which is hardcoded in Dask's distributed scheduler as 3s
. This code path is only hit in Prefect if you're using an existing dask cluster (meaning you pass in an address=
to DaskExecutor
). If you let the DaskExecutor
create and manage its own temporary dask cluster this code path won't be hit at all.Mac Gréco Péralte Chéry
09/15/2020, 2:33 PMNuno Silva
09/15/2020, 3:15 PMJim Crist-Harif
09/16/2020, 4:50 PMDASK_DISTRIBUTED__COMM__TIMEOUTS__CONNECT=20s