https://prefect.io logo
a

Adam Brusselback

02/09/2021, 8:10 PM
One thing I am also not quite clear on, is what the real benefit of stored_as_script is? What does it buy you that pickling doesn't?
j

Jim Crist-Harif

02/09/2021, 8:11 PM
It lets you edit the source to make code (but not structural) changes post registration without requiring re-registration. It also is more flexible than a pickled flow - a pickled representation isn't guaranteed to load in environments running with different versions of python/prefect. But we try hard not to break (without warning) the user-facing apis, so a script-based flow should be loadable across a wider range of environments.
It also lets you put the source there yourself (e.g. upload to github/s3/etc...). pickle-based-flows must be stored somewhere by prefect itself, which means you have less control over the upload process.
Does that make sense?
b

Billy McMonagle

02/09/2021, 8:24 PM
If I may piggyback onto this question... what about
stored_as_script
in the context of docker storage?
a

Adam Brusselback

02/09/2021, 8:25 PM
It does make sense, changing some non-structural code without requiring registration does seem useful for debugging, as does the greater compatibility across environments. Just kinda confusing as to what all is affected when going one way or another though.
j

Jim Crist-Harif

02/09/2021, 8:26 PM
If I may piggyback onto this question... what about
stored_as_script
in the context of docker storage?
That seems less useful IMO. I could see a use case where you manually add the script files yourself in a
Dockerfile
, but that's a rare use case. We probably only added that for parity with the other storage classes.
1
Just kinda confusing as to what all is affected when going one way or another though.
Totally understandable. I'll make a note to expand our storage docs to provide more info here.
b

Billy McMonagle

02/09/2021, 8:30 PM
I've always found it confusing too, thanks for your reply.
a

Adam Brusselback

02/09/2021, 8:35 PM
So if my flow is stored as a script, and deployed manually to N servers with prefect agent local running, is it possible to use local storage for that flow and have it work with multiple agents?
j

Jim Crist-Harif

02/09/2021, 8:38 PM
Sure, provided the script is available at that path on all machines. By default local storage also adds a hostname label marking the flow to only run on the host it was registered with. Local agents do the same. You can disable this behavior though, which you'd likely need to for the above setup:
Copy code
flow.storage = Local(..., add_default_labels=False)

# and for starting the agent via cli
prefect agent local start --no-hostname-label
a

Adam Brusselback

02/09/2021, 8:39 PM
where the heck did I miss the add_default_lables option for local storage... I was explicitly setting the lables as None on the run_config and even that wasn't working.
j

Jim Crist-Harif

02/09/2021, 8:40 PM
it's the only storage/agent that does this. everything else is opt-in only (the reason local storage does this is historical, and may be something we remove someday).
a

Adam Brusselback

02/09/2021, 8:41 PM
Hmm, don't see it mentioned anywhere on that page (ctrl + f)
j

Jim Crist-Harif

02/09/2021, 8:41 PM
Under "automatic labels"
oh, but no description of how to remove it. hmmm.
Ah, it was just added, but the docs are still pinned to the previous release. Updated section will be out later today with the next release.
a

Adam Brusselback

02/09/2021, 8:42 PM
Yeah, had no idea that was an argument I could pass in. Was looking all over thinking I was going crazy
Ah! Makes sense
j

Jim Crist-Harif

02/09/2021, 8:43 PM
github renders this weird (vuepress custom markdown), but the content is here: https://github.com/PrefectHQ/prefect/blob/master/docs/orchestration/flow_config/storage.md#local
a

Adam Brusselback

02/09/2021, 8:44 PM
Thank you very much for the help here!
👍 1