Jacob Longhurst
07/27/2022, 3:02 AMprefecthq/prefect:1.1.0-python3.8
.
If we use the DaskExecutor
with a custom class and kwargs, we get an error message saying we need to install kubectl. If we choose to use a LocalDaskExecutor
it works just fine. It seems it’s failing some where around attempting to talk between the dask scheduler and workers. We’ve created the roles and role bindings dask requires in order to function properly and we’re not sure if we need to pass in a dask scheduler spec and / or service spec to the dask KubeCluster object?
Hoping someone has been able to solve running a temporary, on demand distributed dask cluster in K8S with the KubernetesRun prefect run configAnna Geller
07/27/2022, 2:16 PM