https://prefect.io logo
i

Ievgenii Martynenko

05/06/2022, 8:16 AM
Investigating S3 storage a bit deeper. If we register Flow with S3 and 'local_script_path' set, Prefect copies dataflow to S3 named with specific timestamp; if we register new version, then new file is created. In this way we achieve dataflow versioning. If we use only 'key' option of S3 storage, then we can keep our own names of files, but have to copy dataflow manually. Registering new version of dataflow still increases version in UI, but in fact you have the same dataflow file on S3. So versioning doesn't actually work, right?
a

Anna Geller

05/06/2022, 10:00 AM
Is your issue about flow object versioning or object storage file/data versioning or flow code versioning in general? • Prefect automatically versions your flows - anytime you register a flow, Prefect will bump up the flow version, as long as flow metadata changes. For more nuances about that, check https://discourse.prefect.io/t/when-do-i-need-to-reregister-my-flow-which-changes-in-flow-metadata-result-in-prefect-bumping-up-the-flow-version/403 • For versioning of flow code, you can commit your flow code to a git repo. • For versioning of files/data, you can turn on versioning on your S3 bucket which can help with disaster recovery and troubleshooting Not sure if this is helpful to you, but I thought looking at versioning in those categories can be helpful regarding upload on registration - yes - if you set key and local_script_path, Prefect will upload this local file to the given S3 key (path) during registration - see e.g. this simple example
Registering new version of dataflow still increases version in UI,
The Discourse topic shows how you can avoid that
❤️ 1
i

Ievgenii Martynenko

05/06/2022, 10:32 AM
Thanks, you pointed me to right search direction.
👍 1
31 Views