https://prefect.io logo
a

Aiden Price

12/02/2020, 10:25 PM
Hi everybody! I'm a bit confused by the new
run_config
I can get
KubernetesRun()
to work with no problems at all. But I'm unsure which
run_config
to use for my pre-existing Dask cluster. I presume I need the
DaskExecutor
but do I use a
LocalRun
with it like I used to use a
LocalEnvironment
? I'm a fan of this change by the way. Thank you all.
j

josh

12/02/2020, 10:27 PM
Hey @Aiden Price the dask executor you use is completely independent of the run config! You can now set the executor like so:
Copy code
from prefect.engine.executors import DaskExecutor

flow.executor = DaskExecutor(...)

flow.run_config = KubernetesRun(...)
a

Aiden Price

12/02/2020, 10:28 PM
Thanks Josh, I'm still a bit mixed up here, wouldn't that code create a Kubernetes Job and execute on the Dask cluster?
j

josh

12/02/2020, 10:30 PM
Yes it will. The Kubernetes job is creates is the resources needed to start the flow and the executor is where the tasks are actually executed. So in previous versions where you would have:
Copy code
flow.environment = LocalEnvironment(
    executor=DaskExecutor("address_here")
)
that executor is now broken up into its own field on the flow object. This performed similarly where the k8s agent would create a job and then that job would still execute the tasks on the dask cluster. If you’re using a Kubernetes agent then you should use a
KubernetesRun
and subsequently if you’re using a Local agent you should use a
LocalRun
a

Aiden Price

12/02/2020, 10:35 PM
Okay, the approach I was working towards was actually to have two agents running, a Local agent for flows destined for the Dask cluster and a Kubernetes agent for Jobs and then sort flows to one or the other by labels. But that may be more trouble than needed.
I'm trying to cut the cold start time to as small as possible for a couple of high frequency jobs.
b

Braun Reyes

12/02/2020, 11:02 PM
lol I was about to post a question about this since we also use local env and dask executor for all our flows. With the run config we can let the developer decided if they want to use local dask cluster or not. We were injecting it for them. so dope!
sound like the executor will cloud provider work then if you want remote ecs workers
I can ask Jim seems like folks using the DaskCloudProviderEnvironment would need something comparable with the run config stuff
j

Jim Crist-Harif

12/03/2020, 1:33 AM
Sorry all, just saw this thread. I mentioned to Braun earlier, but I just pushed a docs section (not merged) on this: https://github.com/PrefectHQ/prefect/pull/3754
a

Aiden Price

12/03/2020, 1:53 AM
Looks like excellent advice, thank you @Jim Crist-Harif