What strategies are folks using for flow storage blocks in 2.0? It seems like there are at least two distinct possibilities: 1. Save all flows using one storage block and running the

deployment

build

and

apply

commands from a parent directory containing multiple flows. This way, we use only one storage block but if multiple developers are working in the same shared bucket like in a staging environment, the blast radius is bigger. For example, if a developer is working on only a single flow, there is a risk of overwriting someone else’s in-development code if you commit code from another flow/folder accidentally.

deployment build flow_a/flow.py:entry_point -n flow_a_deployment --storage-block s3/universal-storage-block

flows/ <working directory>
   flow_a/
   flow_b/

2. Every flow gets its own storage block and running

deployment

build

and

apply

commands from that flow’s root directory. This will obviously require us to use more storage blocks, but seems to decrease blast radius.

deployment build ./flow.py:entry_point -n flow_a_deployment --storage-block s3/individual-storage-block

flow_/a <working directory>

It seems to me like option 2 is more optimal, but are there any disadvantages or limitations we should be aware of in using multiple storage blocks?

we are working on introducing 2 things that may help you here soon 1. Multi-flow build -- will look as follows:

prefect deployment build --path flows/flow1.py:one --path flows/flow2.py:two ...

2. Docker storage build - i.e. flow code baked into the container for all flows

That’s great news, thanks for sharing Anna! We’ve found that separating flows from the rest of the image is a really awesome 2.0 feature and enables our CD processes to run much quicker! Regarding multi-flow builds, is the idea here that the

--path

argument would only save that particular path to storage, even if the

deployment build

command is running from a parent directory? In other words, would this allow us to only selectively save flows that all live in different folders, under a common parent folder and using a single storage block, without overwriting all the other flows?

@Ilya Galperin Great question! I have the same question and the way we are getting around this is by creating separate blocks for each deployment to separate code in our remote storage.

Thanks for sharing John! Have you ran into any issues/unexpected side effects with “block proliferation”? Will be very curious to see if multi-flow build that Anna described here is able to address this issue!

Definitely! Lots of blocks, but easier to manage the blocks vs. messy remote deployment blobs/folders!