Can anyone explain the correspondence of KubernetesRun and pods that get spun up. Is KubeRun what the Agent runs to create a scheduler pod? Does it have anything to do with the worker environment?
k
Kevin Kho
09/01/2021, 2:28 AM
KubernetesRun since up a job. This is the client. DaskExecutor spins up the scheduler and workers so the Kubernetes image is independent from the DaskExecutor because it spins that up separately.
With Dask though, the three are related and often need matching packages. My understanding is that the client sends work to the scheduler, which sends work to workers, so all 3 would ideally need similar versions.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.