Sam Werbalowsky
09/30/2021, 8:34 PMPREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS
with dask-gateway? We use a dynamic cluster per flow, rather than static, so we can’t fill in an exact address, as it depends on the cluster that has spun up as part of the flow running.
My goal is to have a helper file that configures the default executor and runconfig for a user so they don’t have to worry about their flow’s configuration.Kevin Kho
09/30/2021, 8:38 PMFlow
and then add their executor and runconfig so they don’t have to repeatedly do it. Maybe you can do the same but have it take inputs?. If you have a function, you can also set it that way?
What issue are you running into? If you feel that the executor is not being respected, it may very well be because the executor is not serialized when a flow is registered. The executor is looked for in the flow file when the flow is loaded from storage. Because of this, it can not be set properly if the executor is not in the file.Sam Werbalowsky
09/30/2021, 8:43 PMclass MyFlow(Flow):
self.config = config()
def config():
if LOCAL_DEVELOPER_MACHINE:
return LocalExecutor()
else:
gateway = Gateway()
cluster = gateway.new_cluster()
return DaskExecutor(cluster.stuff)
I figured I could avoid this by just using the defaults.from helpers import MyFlow
with MyFlow() as flow:
...
Kevin Kho
09/30/2021, 8:53 PMSam Werbalowsky
09/30/2021, 9:01 PM