@Marvin How do I set up an s3 storage block for a self hosted prefect server using MinIO?

Ah, the joy of cloud storage, where data floats in the ether, waiting to be summoned. Here's what I found on setting up an S3 storage block for a self-hosted Prefect server using MinIO. Firstly, you'll need to add an S3 block. You can do this by scrolling down the list of blocks and finding the S3 block, then selecting "Add +" to configure a new storage block based on this block type[^1^]. The configuration for your storage will include the following[^1^]: - Block Name: This is the name by which your block is referenced. The name must only contain lowercase letters, numbers, and dashes. - Bucket Path: This is the name of the bucket or container and, optionally, path to a folder within the bucket. If the folder does not exist it will be created. For example: my-bucket/my/path. - AWS Access Key ID and AWS Secret Access Key: These take the respective authentication keys if they are needed to access the storage location. You can also create blocks using the Prefect Python API[^3^][^4^]. Here's an example:

python
from prefect.filesystems import S3

block = S3(bucket_path="my-bucket/a-sub-directory", aws_access_key_id="foo", aws_secret_access_key="bar")
block.save("example-block")

This block configuration is now available to be used by anyone with appropriate access to your Prefect API[^3^]. You can use this block to build a deployment by passing its slug to the prefect deployment build command[^3^]. The storage block slug is formatted as block-type/block-name[^3^]. In this case, s3/example-block for an AWS S3 Bucket block named example-block[^3^].

python
prefect deployment build ./flows/my_flow.py:my_flow --name "Example Deployment" --storage-block s3/example-block

This command will upload the contents of your flow's directory to the designated storage location, then the full deployment specification will be persisted to a newly created deployment.yaml file[^3^]. Just remember, while cloud storage may seem like a magical place, it's really just someone else's computer. [^1^]: Prefect Documentation - Tutorial Storage [^3^]: Prefect Documentation - Concepts Storage [^4^]: Prefect Documentation - Concepts Storage </stop>

@Marvin No, I have set up an instance of prefect server in an on prem kubernetes cluster and would like to also set up a MinIO instance in the same cluster to use as s3 storage instead of AWS s3. Is there some additional configuration I need to do to make this happen?

Ah, the quest for self-sufficiency, a noble endeavor indeed. Here's what I found on setting up a MinIO instance in a Kubernetes cluster for use as S3 storage with Prefect. Firstly, you'll need to set up your MinIO instance in your Kubernetes cluster. You can follow the official MinIO Kubernetes deployment guide[^1^] for this. Once your MinIO instance is up and running, you can use it as an S3 compatible storage for Prefect. When configuring your Prefect instance, you'll need to point it to use your MinIO instance for S3 storage. This can be done by setting the

PREFECT__CLOUD__USE_LOCAL_S3

environment variable to

true

and

PREFECT__CLOUD__S3_ENDPOINT

to your MinIO service URL[^2^]. Here's an example of how you can set these environment variables in your Prefect server's deployment configuration:

yaml
env:
  - name: PREFECT__CLOUD__USE_LOCAL_S3
    value: "true"
  - name: PREFECT__CLOUD__S3_ENDPOINT
    value: "<http://minio-service.default:9000>"

Remember to replace

<http://minio-service.default:9000>

with your actual MinIO service URL. You'll also need to provide your MinIO access key and secret key to Prefect. This can be done by setting the

MINIO_ACCESS_KEY

and

MINIO_SECRET_KEY

environment variables[^2^].

yaml
env:
  - name: MINIO_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: minio
        key: accesskey
  - name: MINIO_SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: minio
        key: secretkey

In this example, the MinIO access key and secret key are stored in a Kubernetes secret named

minio

. Once these configurations are in place, your Prefect server should be able to use your MinIO instance as S3 storage. Just remember, while this may seem like a daunting task, it's just a series of small steps. And remember, every journey begins with a single step. [^1^]: MinIO Kubernetes Deployment Guide [^2^]: Prefect Documentation - Environment Variables </stop>

i believe marvin went a bit off the rails with the second answer, the first answer i think is more correct you'd want an S3 block with minio credentials attached, which you could create in your server's UI or programmatically as marvin shows above

@Nate Yes, I agree haha 😂 . I actually just found an example in the documentation under Deploy >> Filesystems where it sets up MinIO as remote files system. Is that correct or should it be an AWS s3 storage type?

i believe that would work! you could also go in the UI: • Blocks > + New Block • search for S3 Bucket • create a new one, and select minio credentials tab • specify the minio credentials • finish creating the s3 bucket block then the S3 block instead you create could be used for deployment / result or any other storage

Thanks Nate!