Tom Forbes06/15/2021, 1:11 PM
obviously the second DaskExecutor won’t work locally, which is a shame. I can’t find any supported way of checking if we’re serializing this for production rather than debugging it locally
if running_locally: flow.executor = LocalDaskExecutor() else: flow.executor = DaskExecutor(cluster_class=lambda: KubeCluster(pod_template=....))
Kevin Kho06/15/2021, 1:54 PM
when you say running locally? Can't you change the executor under a
if __name__ == "__main__":
Tom Forbes06/15/2021, 2:06 PM
is one way, but it feels a bit unwieldily. We where planning on just having this interface:
from internal_library import get_executor with Flow(executor=get_executor()): ...
return a different class depending on the context.
Kevin Kho06/15/2021, 2:09 PM
Tom Forbes06/15/2021, 2:18 PM
but this kind of sucks as we need to do the same thing for result classes.
from internal_library import k8s_executor, local_executor with Flow(executor=k8s_executor()) as flow: ... if __name__ == "__main__": flow.run(executor=local_executor())
Zanie06/15/2021, 3:44 PM
when the flow is loaded from storage since 0.14.20
Tom Forbes06/15/2021, 3:47 PM
is something I tried, but it’s only available at runtime (which makes sense). I’ll use it to grab the
class instead of fiddle with the executor