Tom Forbes
06/15/2021, 1:11 PMif running_locally:
flow.executor = LocalDaskExecutor()
else:
flow.executor = DaskExecutor(cluster_class=lambda: KubeCluster(pod_template=....))
obviously the second DaskExecutor won’t work locally, which is a shame. I can’t find any supported way of checking if we’re serializing this for production rather than debugging it locallyKevin Kho
06/15/2021, 1:54 PMflow.run()
when you say running locally? Can't you change the executor under a if __name__ == "__main__":
?Tom Forbes
06/15/2021, 2:06 PMflow.run()
is one way, but it feels a bit unwieldily. We where planning on just having this interface:
from internal_library import get_executor
with Flow(executor=get_executor()):
...
and have get_executor()
return a different class depending on the context.get_local_executor()
🤷Kevin Kho
06/15/2021, 2:09 PMTom Forbes
06/15/2021, 2:18 PMfrom internal_library import k8s_executor, local_executor
with Flow(executor=k8s_executor()) as flow:
...
if __name__ == "__main__":
flow.run(executor=local_executor())
but this kind of sucks as we need to do the same thing for result classes.Zanie
06/15/2021, 3:44 PMprefect.context.get("running_with_backend")
will be True
when the flow is loaded from storage since 0.14.20flow_run_id
ie prefect.context.get("flow_run_id")
Tom Forbes
06/15/2021, 3:47 PMrunning_with_backend
is something I tried, but it’s only available at runtime (which makes sense). I’ll use it to grab the Result
class instead of fiddle with the executor