<https://docs.prefect.io/api/latest/environments/s...
# prefect-community
s
https://docs.prefect.io/api/latest/environments/storage.html#local <- is
stored_as_script
and
path
keyword arguments removed? in every storage class? (
Local
in my case)
j
Hi @Sven Teresniak these kwargs are currently in the master branch and will be available in the next release coming today!
s
Wow nice! 🙂 Is it possible for you to postpone the release until https://github.com/PrefectHQ/prefect/pull/3005 is reviewed and merged?
Sadly, the
i-key
bug (which is solved by this PR) is blocking me a bit. 🙂
j
I’m reviewing that as we speak 🙂
s
I love you guys 🙂
j
We love you too ❤️
s
Sorry, just one more question. The documentation about
path
is not absolutely clear. WHEN do I want to use
path
and what if I specify both,
directory
and
path
?
j
Ah yeah that could be cleared up.
path
is saying “Hey my flow script lives here (it is a direct path) or will live here” and
directory
is still used in the pickle-based storage to say “Store my flow pickles in this directory”
s
At the moment I'm migrating a large ETL system based on a lot of python scripts and a PostgresDB towards Presto+Spark. Prefect will be the glue between all pieces of the puzzle. Solving the i-key-bug after 24h will make a very good impression. Its not that easy to add new tech to our stack (although we have a very modern infrastructure based on K8S, Spark3, Presto, Alluxio etc.)
keep up the good work! 🙂
Yeah, okay, but when I
register()
a flow, why do I need the path to the flow?
I can either call flow.register() using python (executable) or prefect CLI like
prefect register …
. Both will pickle the script in put it under
directory
. Why path?
j
File-based flow storage takes a bit of a different approach compared to the pickle-based method. Where when pickling up the flow we would build and store it on register, the file-based storage actually separates out the two functions. File-based storage now runs with the idea that you place your flow script anywhere you want (S3, Local, GitHub, etc.) and when you register you actually say on the flow’s storage “My flow lives at this path location in this S3 bucket or this GitHub repo etc.)
directory
will only be used (and pickle storage) if you have
stored_as_script=False
(which is the default)
s
Aaaah now I understand. Its now possible to register flows using (for example) a py-file with just one single storage-definition (with path and stored_as_scripts=True) in it
j
Yep!
s
cool thanks. please drop a line in one of the slack channels when the release is out. 👍
j
@Sven Teresniak
0.12.5
has been released
s
thanks. i will deploy the new version this morning 🙂
deployed. fix confirmed. 🙂 thanks