Hi everybody! I'm a bit confused by the new

run_config

I can get

KubernetesRun()

to work with no problems at all. But I'm unsure which

run_config

to use for my pre-existing Dask cluster. I presume I need the

DaskExecutor

but do I use a

LocalRun

with it like I used to use a

LocalEnvironment

? I'm a fan of this change by the way. Thank you all.

Hey @Aiden Price the dask executor you use is completely independent of the run config! You can now set the executor like so:

from prefect.engine.executors import DaskExecutor

flow.executor = DaskExecutor(...)

flow.run_config = KubernetesRun(...)

Thanks Josh, I'm still a bit mixed up here, wouldn't that code create a Kubernetes Job and execute on the Dask cluster?

Yes it will. The Kubernetes job is creates is the resources needed to start the flow and the executor is where the tasks are actually executed. So in previous versions where you would have:

flow.environment = LocalEnvironment(
    executor=DaskExecutor("address_here")
)

that executor is now broken up into its own field on the flow object. This performed similarly where the k8s agent would create a job and then that job would still execute the tasks on the dask cluster. If you’re using a Kubernetes agent then you should use a

KubernetesRun

and subsequently if you’re using a Local agent you should use a

LocalRun

Okay, the approach I was working towards was actually to have two agents running, a Local agent for flows destined for the Dask cluster and a Kubernetes agent for Jobs and then sort flows to one or the other by labels. But that may be more trouble than needed.

lol I was about to post a question about this since we also use local env and dask executor for all our flows. With the run config we can let the developer decided if they want to use local dask cluster or not. We were injecting it for them. so dope!

Sorry all, just saw this thread. I mentioned to Braun earlier, but I just pushed a docs section (not merged) on this: https://github.com/PrefectHQ/prefect/pull/3754

Looks like excellent advice, thank you @Jim Crist-Harif