https://prefect.io logo
Title
s

Seth Coussens

08/26/2022, 8:55 PM
Anybody have anyluck with a storage block that you can use for multiple flows that doesn't just write over your files when you build the deployment?
1
i

Ilya Galperin

08/26/2022, 8:55 PM
Hey Seth - we overcome this by having one storage block per flow and having each storage block point to a different folder inside a remote bucket.
s

Seth Coussens

08/26/2022, 8:56 PM
Do you use any form of CI/CD?
i

Ilya Galperin

08/26/2022, 8:56 PM
Yes
s

Seth Coussens

08/26/2022, 8:56 PM
I guess you just have a script to create the block for the flow then?
i

Ilya Galperin

08/26/2022, 8:56 PM
We create the block using Python i.e.
def create_storage_block(bucket: str, flow: str) -> str:
    path = flow.replace("_", "-")
    block_name = f"{path}-code"

    AWS_ACCESS_KEY_ID = "XYZ"
    AWS_SECRET_ACCESS_KEY = "XYZ"

    block = S3(
        bucket_path=f"{bucket}/flows/{path}",
        aws_access_key_id=AWS_ACCESS_KEY_ID,
        aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    )
    block.save(block_name, overwrite=True)
    print(f"Storage block {block_name} created.")
🙌 1
ya exactly
s

Seth Coussens

08/26/2022, 8:57 PM
Thank you sir, I'll use this while we wait for the github SB.
👍 1
a

Anna Geller

08/26/2022, 8:57 PM
this got merged https://github.com/PrefectHQ/prefect/pull/6550 and will allow you to upload your project only once and skip upload for all subsequent deployments that share the same project dir
s

Seth Coussens

08/26/2022, 8:58 PM
That doesn't seem to cover the same issue though, as that would just mean you don't overwrite the code you already put there.
🙌 1
a

Anna Geller

08/26/2022, 8:58 PM
release ETA is next week
we'll add docs clarifying how to use that in our PDF https://discourse.prefect.io/t/prefect-deployments-faq/1467
❤️ 1
s

Seth Coussens

08/26/2022, 8:59 PM
That one looks like it would solve the issue.
🙌 1
Assuming you can use it like an override or just when specifying the SB so you could add 'subdirectories' to the blocks root.
a

Anna Geller

08/26/2022, 9:02 PM
yup, tons of flexibility here, we'll probably need some Discourse topic on how to not get overwhelmed over all those possibilities
s

Seth Coussens

08/26/2022, 9:03 PM
I think once this is solved, that's my last hurdle for conversion. I've got all the docker and custom package stuff, including the CI/CD figured out.
Even figured out how to use Poetry to load packages into the docker image at prefect build time so we don't need a custom base image.
Custom packages from private Git.
Once I get it all nailed down I'll post something on Discourse.
:thank-you: 2
💯 2
a

Anna Geller

08/26/2022, 9:04 PM
nice! if you would like to share it in a blog post or Discourse topic or even share a GitHub repo, that would be awesome!
👍 1