Thread
#prefect-community
    a

    Andrey Tatarinov

    1 year ago
    What are the best practices to mix local development and remote execution? I have a flow which operates with relatively large chunks of data: 100+Mb XML files. I would like to iterate over solution locally with local caching enabled via LocalResult, and then submit flow to K8s environment which would use GCSResult to persist in the cloud. What would be the best way to organize my code to achieve that behaviour?
    a

    Andrew Hannigan

    1 year ago
    The agent is what determines which execution environment you run on. So you should be able change execution environments without any change to your code. Just send the job to a kubernetes agent instead of the default local agent: https://docs.prefect.io/orchestration/tutorial/k8s.html#running-a-kubernetes-agent
    a

    Andrey Tatarinov

    1 year ago
    @Andrew Hannigan does it mean that I can define my task @task(result=LocalResult(...)) def f()... And later somehow redefine the way results of this specific task are handled?
    a

    Andrew Hannigan

    1 year ago
    I might suggest using cloud-native storage even when running a local agent for this reason
    a

    Andrey Tatarinov

    1 year ago
    I see your point. In my case, I would prefer to keep development and production separate
    We ended up with a function
    def make_flow(Result)
    which accepts Result constructor.
    a

    Andrew Hannigan

    1 year ago
    Makes sense - The result location can be formatted dynamically based on the prefect context. So if you wanted you could also pass in an EXECUTION_ENV environment variable to the prefect ccontext, which you could use to format your result location dynamically. But what you've described sounds like it will also work fine.