Hey team - whats the best practice for building a ...
# prefect-communityb
Brad
08/11/2022, 12:21 AMHey team - whats the best practice for building a deployment for a package with multiple flows and pushing to a storage ie s3?
Brad
08/11/2022, 12:21 AMcurrently I have a python package with a few flows, so I build a deployment for each and apply, but I end up uploading the package data to my storage block multiple times
Brad
08/11/2022, 12:23 AMand is it possibly to push to a subpath of the storage block?
Say I have a storage block in s3:
<s3://brad/flows>flow1flow2Brad
08/11/2022, 12:24 AMAt the moment it seems like I would need to create a storage block per flow (otherwise they will clobber each other)? Is that correct
p
Pipat (Benz) Methavanitpong
08/11/2022, 3:06 AMI'm also noticing this weird behaviour. Building a deployment uploads all files under its current directory, so unrelated flows also get overwritten.
One workaround is to create a block for each developer and environment, and let them manage their storage on their own to avoid overwriting others' files.
b
Brad
08/11/2022, 3:13 AMyeah - but then its one block per flow right @Pipat (Benz) Methavanitpong ? TRhats not going to scale well (I store all my flows in a single repo)
p
Pipat (Benz) Methavanitpong
08/11/2022, 3:18 AMI meant a block per developer, and a developer uploads a whole repo into his block. Because we can't avoid uploading unrelated flows in a deployment.
This should make accidentally overwriting other flows more manageable.
c
Chris L.
08/11/2022, 8:31 AM@Brad I think it makes more sense to have one infra block (e.g. KubernetesJob block) per Dockerized environment and one storage block (e.g. S3 block) per branch (e.g. main / staging) per repo
Chris L.
08/11/2022, 8:33 AMThen multiple flows with the same Docker image in the same Python subpackage can be associated with 1 infra block and 1 storage block accordingly.
b
Brad
08/11/2022, 9:25 PMI agree @Chris L. - but if I want to generate a deployment yaml for each flow, then I end up re-uploading the build artifact every time right?
c
Chris L.
08/12/2022, 1:07 AMYeah…but hopefully the new local storage for Docker resolves this issue. Or maybe something similar to git storage in 1.0
b
Brad
08/12/2022, 7:51 AMyeah git storage would for sure solve my issue
5 Views