Hi, I’m trying to hook up a <min.io> tenant runnin...
# prefect-community
s
Hi, I’m trying to hook up a min.io tenant running on a k8s as a storage block. Is that supported at the moment? Seems the aws s3-bucket doesn’t have get/put-directory?
1
j
Hi Sander. Check out the prefect-aws collection: https://github.com/PrefectHQ/prefect-aws
s
I did look at the s3-bucket from prefect-aws but for some reason I see that it doesn’t support get/put-directory needed for a storage block?
Hmm. Seems I was looking at the online manual only. Will check this out.
@Jeff Hale do you know when this PR is expected to be merged?
j
I don’t, but @alex and @Andrew Huang might. 🙂
🚀 1
a
we’ll be working on it this week!
👍 1
🚀 1
🎉 1
s
Looking forward to the merge:)
👀 1
a
This is released now.
🦜 3
s
I was actually just checking:) I can confirm indeed that it registers nicely as a storage block. Thanks!
🙌 1
Running into other stuff now.
a
let me know if we can help
s
KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."
Trying to run my flow as a kubernetes job.
I think prefect version in my kubernetes cluster does have the latest aws lib.
a
This happens when block documents are using an old schema which does not exist in code anymore. I think
I would try re-registering
prefect block register -m prefect_aws
and then recreate the block
s
Let me redo that cycle now
a
(there may be a better way if anyone else knows)
a
@Sander Can you verify that
prefect-aws
is installed in the container running as part of your Kubernetes job?
s
Yeah. I was thinking that could be the issue.
This is the error log generated by the kubernetesjob pod.
Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run flow = await load_flow_from_flow_run(flow_run, client=client) File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 153, in load_flow_from_flow_run storage_block = Block._from_block_document(storage_document) File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 540, in _from_block_document else cls.get_block_class_from_schema(block_document.block_schema) File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 571, in get_block_class_from_schema return lookup_type(cls, block_schema_to_key(schema)) File "/usr/local/lib/python3.10/site-packages/prefect/utilities/dispatch.py", line 186, in lookup_type raise KeyError( KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."
Hard to copy paste nicely on my phone:)
Trying to find the kubernetesjob manifest
I only see the prefect-job container there.
I think I need to manually install as a customisation. How should I go about that?
a
can you elaborate what you mean by customisation? is this a Kubernetes term?
s
What I think is happening is that the latest prefect_aws lib is not in the prefect-job container that is used by default in the kubernetes job manifest.
I see that I can customise the manifest in the kubernetesjob block.
However , at the moment I’m not sure how to make sure I update the appropriate libs (pip install prefect_aws in the prefect-job container when it is started as part of the kubernetesjob
a
s
Will check it out and get back. Thanks!
I think that will help indeed. Will try out EXTRA_PIP_PACKAGES
a
I’m glad! thanks for checking it out
s
Copy code
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.
I think I need to figure out a bit about my minio setup
Ok getting somewhere.
Copy code
botocore.exceptions.SSLError: SSL validation failed for <https://minio:443/k8s?prefix=&encoding-type=url> [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
@Andrew Huang would you happen to know if and how I can disable SSL verification when connecting to my local minio tenant running in my local kubernetes cluster?
a
Is SSLVerification set on boto3 config? if so, this might be related https://github.com/PrefectHQ/prefect-aws/issues/124
s
I’m inclined to think that I need to do some plumbing on my cluster but I’ll definitely read that link. (A bit long to get the gist now :) ) thanks!
👀 1
Having read the link I’m not sure it discusses the same issue. I’ll have to investigate how the SSL auth works on the minio tenant inside my k8s first I guess.
a
Copy code
s3 = boto3.client('s3', verify=False)
it might be related because there’s no way to pass verify=False atm
s
Oh. Btw. The extra pip packages did the job! Thanks
👍 1
cc @alex
s
Ah in that sense. No indeed. Then it’s likely related. Nevertheless, I think the kubernetes cluster can auto issue certificates
But not sure.
Now I see. Passing these options to the s3 bucket should definitely solve this SSL issue.
Hi @alex and @Andrew Huang I tried to look at changing the code myself, but I think I’m missing some background knowledge to properly help out here. How can we move this missing feature forward as currently I would need to jump quite some hoops to set up ssl within my cluster and I rather be able to not verify. Is there a solution thinkable where we keep the property issue discussed in the links for later and for now provide a means to avoid the ssl verification? What are your thoughts? Thanks!
👀 1
Let me double check if all was set correctly on my minio tenant.
Just double checked and I also get an ssl error when I try to access the tenant via plain python code so I’m not sure if the feature request will solve my issue.
I’ll get back to you once I’m sure this feature is a real blocker for me.
a
when using plain python code, if you disable verify, are you able to get it working?
s
Was trying that route indeed, but then I needed to create a SSL-less tenant which then clashed with some ports. So need to work more to better understand the setup. What do surprises me though is that I don’t get any complaints from prefect server when I create a manifest with my minio storage block and apply the manifest. Also need to look into that so it’s definitely not a clear cut thing yet.
👀 1
Seems I’m getting somewhere but nothing to report yet. Once I have something I’ll let you know.
👍 1
I have been able to write to the minio in the cluster by disabling SSL via an http_client where I put the cert reqs to CERT_NONE.
Current line of approach is either override SSL via env variables or adjust my minio setup.
My hope is that I can set env variables.
I was indeed able to inject the public certificate that is self signed by the minio tenant into the agent pod. This solves the SSL issue. You can use the secret that is generated by the minio operator and save the public.crt string in the secret onto the pod in the standard SSL directory.
1