https://prefect.io logo
Title
s

Sam Werbalowsky

09/30/2021, 8:34 PM
anyone have luck setting
PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS
with dask-gateway? We use a dynamic cluster per flow, rather than static, so we can’t fill in an exact address, as it depends on the cluster that has spun up as part of the flow running. My goal is to have a helper file that configures the default executor and runconfig for a user so they don’t have to worry about their flow’s configuration.
k

Kevin Kho

09/30/2021, 8:38 PM
Hey @Sam Werbalowsky, I haven’t seen anyone try to set this. I have seen some people subclass
Flow
and then add their executor and runconfig so they don’t have to repeatedly do it. Maybe you can do the same but have it take inputs?. If you have a function, you can also set it that way? What issue are you running into? If you feel that the executor is not being respected, it may very well be because the executor is not serialized when a flow is registered. The executor is looked for in the flow file when the flow is loaded from storage. Because of this, it can not be set properly if the executor is not in the file.
s

Sam Werbalowsky

09/30/2021, 8:43 PM
Hey hey, yeah so basically I want to have a library (let’s call it helpers that has something like this…):
class MyFlow(Flow):
    self.config = config()

def config():
   if LOCAL_DEVELOPER_MACHINE:
      return LocalExecutor()
   else:
      gateway = Gateway()
      cluster = gateway.new_cluster()
      return DaskExecutor(cluster.stuff)
I figured I could avoid this by just using the defaults.
and then in each flow just do
from helpers import MyFlow

with MyFlow() as flow:
    ...
k

Kevin Kho

09/30/2021, 8:53 PM
This will work, just that you need helpers available during execution time. I personally haven’t seen the default class set
s

Sam Werbalowsky

09/30/2021, 9:01 PM
Gotcha. Yep, that shouldn't be a problem in our end. Thanks!