Tom Forbes
06/29/2021, 1:59 PMgateway = Gateway()
cluster = gateway.new_cluster()
executor = DaskExecutor(
address=cluster.scheduler_address,
client_kwargs={"security": cluster.security}
)
flow.run(executor=executor)
How is this supposed to work with flows that use Docker storage? The specific executor needs to be resolved at import time, and using the example code in the docs it would mean creating a cluster at import time.Kevin Kho
dask_gateway.GatewayCluster
callable into DaskExecutor
with something like
executor = DaskExecutor(
cluster_class="dask_gateway.GatewayCluster",
cluster_kwargs={
"image": "prefecthq/prefect:latest",
"n_workers": 5,
...
},
)
Kevin Kho
executor = DaskExecutor(
cluster_class=dask_gateway.GatewayCluster,
cluster_kwargs={
"image": "prefecthq/prefect:latest",
"n_workers": 5,
...
},
)
Tom Forbes
06/29/2021, 2:20 PM