Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
pacc-london-2023
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
j
Josh Paulin
08/17/2022, 5:48 PMHello. Starting to look at migrating to Prefect 2.0, and I’m a little confused about how to structure storage blocks for flows. Is it best practice to create a single block for all flows, or to create one block per flow? I’m not seeing any way when building the deployment to specify a subfolder, which makes me think option 1 is kind of forced…
👀 2
o
Oscar Björhn
08/17/2022, 5:52 PMNot sure what the recommendation is, but I tried to make it work with one block at first, and bumped into the exact problem you mention. So now we're just using a block per flow, it's very easy to create the block using Python (or CLI, I suppose) if it doesn't already exist. You don't even need to check if it exists manually, the overwrite option appears to make it safe to create the block whether it already exists or not.
🙌 1
print(f"Creating storage block flows/{azure_block_flow_name}...")
azure_block = Azure(bucket_path=f"flows/{flow_name_clean}", azure_storage_connection_string=connection_string)
azure_block.save(f"azure-{env.value}-{azure_block_flow_name}", overwrite=True)🙌 1
j
Josh Paulin
08/17/2022, 5:57 PMYeah that was probably going to be the road I went down if this was the case. Looking one step ahead as I’m thinking about CI, I’m thinking that’s a reasonable place to try and create the block.
o
Oscar Björhn
08/17/2022, 5:58 PMOh, yeah. I believe you can run an equivalent command using the prefect cli, I just don't know it off the top of my head.
:thank-you: 1
i
Ilya Galperin
08/17/2022, 6:17 PMWe’ve also decided on this approach of 1 storage block per flow for now for what it’s worth. It also seems this is the best way to lessen blast radius if multiple developers are working in the same environment like staging or development.
I asked a similar question here: https://prefect-community.slack.com/archives/CL09KU1K7/p1660264667058509
It does seem like
deployment builda
Anna Geller
08/17/2022, 6:44 PMIs it best practice to create a single block for all flows, or to create one block per flow?you can treat the block that you provide on the deployment as a parent block holding your generic configuration that you may configure once and share across deployments -- when you create a deployment, Prefect copies that configuration and creates anonymous block per deployment
this admittedly long readme file can bring more clarity https://github.com/anna-geller/dataflow-ops
:thank-you: 1
we are working on a feature that will allow multiple flows to be built in one command - this is important for projects with multiple flows and will be critical to prevent re-uploading the entire project directory with every flow's deployment build -- the syntax will most likely look like
prefect deployment build --path ./flows/hello.py:one --path ./flows/other.py:two ..🙌 1