https://prefect.io logo
a

Andrey Tatarinov

12/02/2020, 3:49 PM
What are the best practices to mix local development and remote execution? I have a flow which operates with relatively large chunks of data: 100+Mb XML files. I would like to iterate over solution locally with local caching enabled via LocalResult, and then submit flow to K8s environment which would use GCSResult to persist in the cloud. What would be the best way to organize my code to achieve that behaviour?
a

Andrew Hannigan

12/02/2020, 5:14 PM
The agent is what determines which execution environment you run on. So you should be able change execution environments without any change to your code. Just send the job to a kubernetes agent instead of the default local agent: https://docs.prefect.io/orchestration/tutorial/k8s.html#running-a-kubernetes-agent
a

Andrey Tatarinov

12/02/2020, 6:07 PM
@Andrew Hannigan does it mean that I can define my task @task(result=LocalResult(...)) def f()... And later somehow redefine the way results of this specific task are handled?
a

Andrew Hannigan

12/02/2020, 6:51 PM
I might suggest using cloud-native storage even when running a local agent for this reason
a

Andrey Tatarinov

12/02/2020, 7:44 PM
I see your point. In my case, I would prefer to keep development and production separate
We ended up with a function
def make_flow(Result)
which accepts Result constructor.
a

Andrew Hannigan

12/02/2020, 8:03 PM
Makes sense - The result location can be formatted dynamically based on the prefect context. So if you wanted you could also pass in an EXECUTION_ENV environment variable to the prefect ccontext, which you could use to format your result location dynamically. But what you've described sounds like it will also work fine.
2 Views