Bernardo Galvao
05/09/2022, 11:46 AMAnna Geller
05/09/2022, 11:58 AMBernardo Galvao
05/09/2022, 12:08 PMAnna Geller
05/09/2022, 12:11 PMBernardo Galvao
05/09/2022, 12:12 PMAnna Geller
05/09/2022, 12:12 PMBernardo Galvao
05/09/2022, 12:15 PMprefect storage create
and did try Local Storage. I was hoping to see the directory being created in the filesystem, but I didn't see it happendavzucky
05/09/2022, 12:16 PMAnna Geller
05/09/2022, 12:17 PMdavzucky
05/09/2022, 12:18 PMBernardo Galvao
05/27/2022, 4:23 PMping minio
from the Prefect container alright. โ
โข There are 6 options to pass when creating an S3 storage for Prefect
โข I dont know yet how to validate the BUCKET
option because it accepts any value! So I cannot be sure whether it is working as of yet.
I've created a user "prefect" and a bucket "prefect" from Minio's console. Now I need to know how to pass the BUCKET
value correctly. I find it odd that I cannot point out the S3 endpoint url pointing to my Minio instance before passing the bucket name.
Select a storage type to create: 5
You've selected S3 Storage. It has 6 option(s).
BUCKET: <http://minio/prefect>
REGION NAME (optional):
PROFILE NAME (optional):
AWS ACCESS KEY ID (optional): prefect
AWS SESSION TOKEN (optional):
AWS SECRET ACCESS KEY (optional): prefecty
Choose a name for this storage configuration: minio
Validating configuration...
Registering storage with server...
Registered storage 'minio' with identifier '232d4851-0373-4ecb-b295-18465e88382f'.
root@9bb445098cef:/opt/prefect# prefect storage ls
Configured Storage
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโ
โ โ โ Storage โ โ โ
โ ID โ Storage Type โ Version โ Name โ Server Default โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 232d4851-0373-4ecb-b295-18465e88382f โ S3 Storage โ 1.0 โ minio โ โ
โ 5a6f982f-1b9d-452f-bbc4-9e5a5dc724af โ S3 Storage โ 1.0 โ prefectminio โ ๐ฆ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโ
davzucky
05/30/2022, 12:32 AMBernardo Galvao
05/30/2022, 8:46 AMAnna Geller
05/30/2022, 9:38 AMBernardo Galvao
06/01/2022, 3:44 PMis on-prem the reason why you can't use S3?Yes indeed. The servers sit inside a closed network, so I am resorting to Minio.
Anna Geller
06/01/2022, 8:25 PMBernardo Galvao
06/02/2022, 9:15 AMprefect deployment create
?Anna Geller
06/03/2022, 9:40 AMBernardo Galvao
06/06/2022, 10:08 AM"endpoint_url"
@davzucky here is a solution ๐
References:
โข https://s3fs.readthedocs.io/en/latest/#s3-compatible-storage
โข https://stackoverflow.com/a/71180307/13187430
# for boto
os.environ["AWS_ACCESS_KEY_ID"] = "minio"
os.environ["AWS_SECRET_ACCESS_KEY"] = "miniominio"
minio_storage = FileStorageBlock(
base_path='<s3://prefect/deployments>',
key_type='hash',
options={ # Additional options to pass to the underlying fsspec file system.
# a gamble that as Prefect is using s3fs (that in its turn also uses fsspec)
# these options are consumed by fsspec
"client_kwargs": {"endpoint_url": "<http://minio:9000>"}
}
)
Anna Geller
06/06/2022, 10:43 AM