https://prefect.io logo
Title
j

jack

12/15/2022, 4:14 PM
How do I get each deployment to store its files inside its own directory using S3 block storage?
1
The first deployment I applied pushed its files to the root of the S3 bucket.
c

Christopher Boyd

12/15/2022, 4:19 PM
Hi Jack, you can use relative paths to the storage block
j

jack

12/15/2022, 4:19 PM
Can you provide an example?
c

Christopher Boyd

12/15/2022, 4:21 PM
prefect deployment build ./healthcheck.py:healthcheck -n azure-aci-deploy -t aci -q default -ib azure-container/boydimusprime -sb azure/boydblock/docker_test -a
-sb is the storage block command, azure is the block type, boydblock is the name of the block I created, and docker_test is a sub-path inside the storage (in Azure)
if you’re using s3, it would be like:
-sb s3/block_name/sub-path
j

jack

12/15/2022, 4:24 PM
Using a python module to apply the deployment, where would I put that?
storage = S3.load('flow-storage-block')

deployment = Deployment.build_from_flow(
    flow=jacks_first_flow,
    name='jacks-first-deployment',
    version=2,
    storage=storage,
)

deployment.apply(upload=True)
c

Christopher Boyd

12/15/2022, 4:24 PM
it looks like this is in my storage account:
j

jack

12/15/2022, 4:25 PM
Also, is there an option we can set so that storage is automatically namespaced? (So that different users of the same storage block don't inadvertently use the root of the S3 bucket and end up overwriting each others' files)
c

Christopher Boyd

12/15/2022, 4:25 PM
I’d need to check, I haven’t tested with python
there is not
j

jack

12/15/2022, 4:26 PM
It appears that with prefect v1 each flow was automatically namespaced when stored to S3.
c

Christopher Boyd

12/15/2022, 4:34 PM
I think you’d just add the path=“relative path to root”
Here’s the PR that adds that: https://github.com/PrefectHQ/prefect/pull/6518
j

jack

12/15/2022, 4:43 PM
The
path
keyword is working:
storage = S3.load('flow-storage-block')

deployment = Deployment.build_from_flow(
    flow=jacks_first_flow,
    name='jacks-first-deployment',
    version=2,
    storage=storage,
    path='01-favorite-path',
)

deployment.apply(upload=True)
🙌 2