https://prefect.io
Join Slack
Hey team - whats the best practice for building a deployment for a package with multiple flows and p...
# ask-community
b
Hey team - whats the best practice for building a deployment for a package with multiple flows and pushing to a storage ie s3?
currently I have a python package with a few flows, so I build a deployment for each and apply, but I end up uploading the package data to my storage block multiple times
and is it possibly to push to a subpath of the storage block? Say I have a storage block in s3: 
<s3://brad/flows>
and I want to push 
flow1
and 
flow2
here - is there a way I can push the build artifacts to ``s3://brad/flows/flow1/ and s3://brad/flows/flow2/`
At the moment it seems like I would need to create a storage block per flow (otherwise they will clobber each other)? Is that correct
p
I'm also noticing this weird behaviour. Building a deployment uploads all files under its current directory, so unrelated flows also get overwritten. One workaround is to create a block for each developer and environment, and let them manage their storage on their own to avoid overwriting others' files.
b
yeah - but then its one block per flow right @Pipat (Benz) Methavanitpong ? TRhats not going to scale well (I store all my flows in a single repo)
p
I meant a block per developer, and a developer uploads a whole repo into his block. Because we can't avoid uploading unrelated flows in a deployment. This should make accidentally overwriting other flows more manageable.
c
@Brad I think it makes more sense to have one infra block (e.g. KubernetesJob block) per Dockerized environment and one storage block (e.g. S3 block) per branch (e.g. main / staging) per repo
Then multiple flows with the same Docker image in the same Python subpackage can be associated with 1 infra block and 1 storage block accordingly.
b
I agree @Chris L. - but if I want to generate a deployment yaml for each flow, then I end up re-uploading the build artifact every time right?
c
Yeah…but hopefully the new local storage for Docker resolves this issue. Or maybe something similar to git storage in 1.0
b
yeah git storage would for sure solve my issue
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by