Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
pacc-london-2023
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
b
Brad
08/11/2022, 12:21 AMHey team - whats the best practice for building a deployment for a package with multiple flows and pushing to a storage ie s3?
currently I have a python package with a few flows, so I build a deployment for each and apply, but I end up uploading the package data to my storage block multiple times
and is it possibly to push to a subpath of the storage block?
Say I have a storage block in s3:
<s3://brad/flows>flow1flow2At the moment it seems like I would need to create a storage block per flow (otherwise they will clobber each other)? Is that correct
p
Pipat (Benz) Methavanitpong
08/11/2022, 3:06 AMI'm also noticing this weird behaviour. Building a deployment uploads all files under its current directory, so unrelated flows also get overwritten.
One workaround is to create a block for each developer and environment, and let them manage their storage on their own to avoid overwriting others' files.
b
Brad
08/11/2022, 3:13 AMyeah - but then its one block per flow right @Pipat (Benz) Methavanitpong ? TRhats not going to scale well (I store all my flows in a single repo)
p
Pipat (Benz) Methavanitpong
08/11/2022, 3:18 AMI meant a block per developer, and a developer uploads a whole repo into his block. Because we can't avoid uploading unrelated flows in a deployment.
This should make accidentally overwriting other flows more manageable.
c
Chris L.
08/11/2022, 8:31 AM@Brad I think it makes more sense to have one infra block (e.g. KubernetesJob block) per Dockerized environment and one storage block (e.g. S3 block) per branch (e.g. main / staging) per repo
Then multiple flows with the same Docker image in the same Python subpackage can be associated with 1 infra block and 1 storage block accordingly.
b
Brad
08/11/2022, 9:25 PMI agree @Chris L. - but if I want to generate a deployment yaml for each flow, then I end up re-uploading the build artifact every time right?
c
Chris L.
08/12/2022, 1:07 AMYeah…but hopefully the new local storage for Docker resolves this issue. Or maybe something similar to git storage in 1.0
b
Brad
08/12/2022, 7:51 AMyeah git storage would for sure solve my issue