https://docs.prefect.io/api/latest/environments/storage.html#local <- is

stored_as_script

and

path

keyword arguments removed? in every storage class? (

Local

in my case)

Hi @Sven Teresniak these kwargs are currently in the master branch and will be available in the next release coming today!

Wow nice! 🙂 Is it possible for you to postpone the release until https://github.com/PrefectHQ/prefect/pull/3005 is reviewed and merged?

I’m reviewing that as we speak 🙂

I love you guys 🙂

We love you too ❤️

Sorry, just one more question. The documentation about

path

is not absolutely clear. WHEN do I want to use

path

and what if I specify both,

directory

and

path

?

Ah yeah that could be cleared up.

path

is saying “Hey my flow script lives here (it is a direct path) or will live here” and

directory

is still used in the pickle-based storage to say “Store my flow pickles in this directory”

At the moment I'm migrating a large ETL system based on a lot of python scripts and a PostgresDB towards Presto+Spark. Prefect will be the glue between all pieces of the puzzle. Solving the i-key-bug after 24h will make a very good impression. Its not that easy to add new tech to our stack (although we have a very modern infrastructure based on K8S, Spark3, Presto, Alluxio etc.)

File-based flow storage takes a bit of a different approach compared to the pickle-based method. Where when pickling up the flow we would build and store it on register, the file-based storage actually separates out the two functions. File-based storage now runs with the idea that you place your flow script anywhere you want (S3, Local, GitHub, etc.) and when you register you actually say on the flow’s storage “My flow lives at this path location in this S3 bucket or this GitHub repo etc.)

Aaaah now I understand. Its now possible to register flows using (for example) a py-file with just one single storage-definition (with path and stored_as_scripts=True) in it

Yep!

cool thanks. please drop a line in one of the slack channels when the release is out. 👍

@Sven Teresniak

0.12.5

has been released

thanks. i will deploy the new version this morning 🙂