heyy, I am setting up prefect in an on-prem production environment and I'm looking into setting up Storage. I would like to know if there is any recommended KV store docker image that I should use for this purpose?

The recommended docker image is any of the official Prefect base images here - for Orion, you would need those tagged with 2.0 beta e.g. 2.0b3-python3.9 Regarding KV storage, this is not yet released and documented, so you may need to wait a bit. In the meantime, I'd highly recommend S3 (or similar cloud object storage)

I agree with you, but in my case, the client wants the system to be in a closed private network, which excludes any cloud solution unfortunately. 😞 I would definitely prefer that solution 😄

perhaps S3 on Outposts?

That's nice! I'll loot into it 👀

otherwise local storage

Now that you mention it, I played around with

prefect storage create

and did try Local Storage. I was hoping to see the directory being created in the filesystem, but I didn't see it happen

Did you look at solution like minio or ceph? This allow you to have S3 compatible storage on prem You can look at well at Enterprise grade with dell ECS for example. https://www.dell.com/en-hk/dt/storage/ecs/index.htm#scroll=off We are using a mix of minio and ECS with success with prefect

you should be able to set the desired path and then when you create a deployment, Prefect will store the flow code there

The solution I told you should work out of the box with fsspec.

Hi @davzucky, I am struggling with the final step of configuring Minio storage of Prefect, perhaps you could help me out? • I've spun up a Prefect and a Minio instance in Docker Swarm. • I've

ping minio

from the Prefect container alright. ✅ • There are 6 options to pass when creating an S3 storage for Prefect • I dont know yet how to validate the

BUCKET

option because it accepts any value! So I cannot be sure whether it is working as of yet. I've created a user "prefect" and a bucket "prefect" from Minio's console. Now I need to know how to pass the

BUCKET

value correctly. I find it odd that I cannot point out the S3 endpoint url pointing to my Minio instance before passing the bucket name.

Select a storage type to create: 5
You've selected S3 Storage. It has 6 option(s). 
BUCKET: <http://minio/prefect>
REGION NAME (optional): 
PROFILE NAME (optional): 
AWS ACCESS KEY ID (optional): prefect
AWS SESSION TOKEN (optional): 
AWS SECRET ACCESS KEY (optional): prefecty
Choose a name for this storage configuration: minio
Validating configuration...
Registering storage with server...
Registered storage 'minio' with identifier '232d4851-0373-4ecb-b295-18465e88382f'.
root@9bb445098cef:/opt/prefect# prefect storage ls
                                           Configured Storage                                            
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃                                      ┃              ┃ Storage        ┃              ┃                 ┃
┃                                   ID ┃ Storage Type ┃ Version        ┃ Name         ┃ Server Default  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 232d4851-0373-4ecb-b295-18465e88382f │ S3 Storage   │ 1.0            │ minio        │                 │
│ 5a6f982f-1b9d-452f-bbc4-9e5a5dc724af │ S3 Storage   │ 1.0            │ prefectminio │ 🦆              │
└──────────────────────────────────────┴──────────────┴────────────────┴──────────────┴─────────────────┘

Let me have a try this week, as I haven't done it yet with Prefect 2.0. Will come back to you about that.

thanks 🙂

Thanks so much, davzucky for helping here. Bernanrdo, it could be a Docker network issue - make sure you can access it from where Prefect is running - is on-prem the reason why you can't use S3?

is on-prem the reason why you can't use S3?

Yes indeed. The servers sit inside a closed network, so I am resorting to Minio.

thanks for explaining. Not sure whether you saw the announcement here but we'll release changes to the storage abstraction this Thursday - could you check then whether the issue is resolved?

Yes, I will check as soon as the image is released 🙂

yes exactly, you would need to create a deployment and this way, Prefect will upload your flow to the specified S3 storage

@Anna Geller Made it work with Minio finally 🎉 It was a gamble but it worked 😅 At least within Docker swarm - but probably for any AWS S3 alternative - the solution was to specify

"endpoint_url"

@davzucky here is a solution 🙂 References: • https://s3fs.readthedocs.io/en/latest/#s3-compatible-storage • https://stackoverflow.com/a/71180307/13187430

# for boto
os.environ["AWS_ACCESS_KEY_ID"] = "minio"
os.environ["AWS_SECRET_ACCESS_KEY"] = "miniominio"

minio_storage = FileStorageBlock(
    base_path='<s3://prefect/deployments>',
    key_type='hash',
    options={  # Additional options to pass to the underlying fsspec file system.
        # a gamble that as Prefect is using s3fs (that in its turn also uses fsspec)
        # these options are consumed by fsspec
        "client_kwargs": {"endpoint_url": "<http://minio:9000>"}
    }
)

great work, thanks so much for sharing!