Brennan Tolman
07/06/2023, 6:59 PMMarvin
07/06/2023, 7:00 PMpython
from prefect.filesystems import S3
block = S3(bucket_path="my-bucket/a-sub-directory", aws_access_key_id="foo", aws_secret_access_key="bar")
block.save("example-block")
This block configuration is now available to be used by anyone with appropriate access to your Prefect API[^3^]. You can use this block to build a deployment by passing its slug to the prefect deployment build command[^3^]. The storage block slug is formatted as block-type/block-name[^3^]. In this case, s3/example-block for an AWS S3 Bucket block named example-block[^3^].
python
prefect deployment build ./flows/my_flow.py:my_flow --name "Example Deployment" --storage-block s3/example-block
This command will upload the contents of your flow's directory to the designated storage location, then the full deployment specification will be persisted to a newly created deployment.yaml file[^3^].
Just remember, while cloud storage may seem like a magical place, it's really just someone else's computer.
[^1^]: Prefect Documentation - Tutorial Storage
[^3^]: Prefect Documentation - Concepts Storage
[^4^]: Prefect Documentation - Concepts Storage
</stop>Brennan Tolman
07/06/2023, 7:02 PMMarvin
07/06/2023, 7:03 PMPREFECT__CLOUD__USE_LOCAL_S3
environment variable to true
and PREFECT__CLOUD__S3_ENDPOINT
to your MinIO service URL[^2^].
Here's an example of how you can set these environment variables in your Prefect server's deployment configuration:
yaml
env:
- name: PREFECT__CLOUD__USE_LOCAL_S3
value: "true"
- name: PREFECT__CLOUD__S3_ENDPOINT
value: "<http://minio-service.default:9000>"
Remember to replace <http://minio-service.default:9000>
with your actual MinIO service URL.
You'll also need to provide your MinIO access key and secret key to Prefect. This can be done by setting the MINIO_ACCESS_KEY
and MINIO_SECRET_KEY
environment variables[^2^].
yaml
env:
- name: MINIO_ACCESS_KEY
valueFrom:
secretKeyRef:
name: minio
key: accesskey
- name: MINIO_SECRET_KEY
valueFrom:
secretKeyRef:
name: minio
key: secretkey
In this example, the MinIO access key and secret key are stored in a Kubernetes secret named minio
.
Once these configurations are in place, your Prefect server should be able to use your MinIO instance as S3 storage.
Just remember, while this may seem like a daunting task, it's just a series of small steps. And remember, every journey begins with a single step.
[^1^]: MinIO Kubernetes Deployment Guide
[^2^]: Prefect Documentation - Environment Variables
</stop>Nate
07/06/2023, 7:10 PMBrennan Tolman
07/06/2023, 7:44 PMNate
07/06/2023, 7:57 PMBrennan Tolman
07/07/2023, 3:11 AM