https://prefect.io logo
Title
t

Tom Forbes

06/15/2021, 1:11 PM
How can I customise the flow executor depending on if the flow is being run locally or being serialized to a docker container? i.e I want to do:
if running_locally:
    flow.executor = LocalDaskExecutor()
else:
    flow.executor = DaskExecutor(cluster_class=lambda: KubeCluster(pod_template=....))
obviously the second DaskExecutor won’t work locally, which is a shame. I can’t find any supported way of checking if we’re serializing this for production rather than debugging it locally
k

Kevin Kho

06/15/2021, 1:54 PM
Hey @Tom Forbes, do you mean using
flow.run()
when you say running locally? Can't you change the executor under a
if __name__ == "__main__":
?
t

Tom Forbes

06/15/2021, 2:06 PM
Sure,
flow.run()
is one way, but it feels a bit unwieldily. We where planning on just having this interface:
from internal_library import get_executor

with Flow(executor=get_executor()):
   ...
and have
get_executor()
return a different class depending on the context.
I guess we can just expose another function like
get_local_executor()
🤷
k

Kevin Kho

06/15/2021, 2:09 PM
You're right that this seems to only be doable in script-based storage. I'm not sure a function will work because even if the execution of that is delayed, the executor is already set once that function is run and you won't be able to change it.
t

Tom Forbes

06/15/2021, 2:18 PM
It would work if we documented that you should do:
from internal_library import k8s_executor, local_executor

with Flow(executor=k8s_executor()) as flow: 
    ...

if __name__ == "__main__":
    flow.run(executor=local_executor())
but this kind of sucks as we need to do the same thing for result classes.
But anyway, thanks for helping!
z

Zanie

06/15/2021, 3:44 PM
If you store your flow as a script (vs pickling) this is achievable
prefect.context.get("running_with_backend")
will be
True
when the flow is loaded from storage since 0.14.20
👍 1
Before then, you could check for a
flow_run_id
ie
prefect.context.get("flow_run_id")
t

Tom Forbes

06/15/2021, 3:47 PM
Thanks Michael!
running_with_backend
is something I tried, but it’s only available at runtime (which makes sense). I’ll use it to grab the
Result
class instead of fiddle with the executor