Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
s
Sander
10/15/2022, 8:02 PMHi, I’m trying to hook up a min.io tenant running on a k8s as a storage block. Is that supported at the moment? Seems the aws s3-bucket doesn’t have get/put-directory?
✅ 1
j
Jeff Hale
10/16/2022, 1:06 AMHi Sander. Check out the prefect-aws collection: https://github.com/PrefectHQ/prefect-aws
s
Sander
10/16/2022, 8:33 AMI did look at the s3-bucket from prefect-aws but for some reason I see that it doesn’t support get/put-directory needed for a storage block?
Hmm. Seems I was looking at the online manual only. Will check this out.
Ah. Seems to be a known issue: https://github.com/PrefectHQ/prefect-aws/issues/67
@Jeff Hale do you know when this PR is expected to be merged?
j
Jeff Hale
10/17/2022, 12:31 PMI don’t, but @alex and @Andrew Huang might. 🙂
🚀 1
a
Andrew Huang
10/17/2022, 6:35 PMwe’ll be working on it this week!
👍 1
🚀 1
🎉 1
s
Sander
10/17/2022, 6:36 PMLooking forward to the merge:)
👀 1
a
Andrew Huang
10/19/2022, 8:22 PMThis is released now.
:party-parrot: 3
s
Sander
10/19/2022, 9:09 PMI was actually just checking:) I can confirm indeed that it registers nicely as a storage block. Thanks!
🙌 1
Running into other stuff now.
a
Andrew Huang
10/19/2022, 9:11 PMlet me know if we can help
s
Sander
10/19/2022, 9:12 PMKeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."
Trying to run my flow as a kubernetes job.
I think prefect version in my kubernetes cluster does have the latest aws lib.
a
Andrew Huang
10/19/2022, 9:14 PMThis happens when block documents are using an old schema which does not exist in code anymore. I think
I would try re-registering
prefect block register -m prefect_awss
Sander
10/19/2022, 9:19 PMLet me redo that cycle now
a
Andrew Huang
10/19/2022, 9:20 PM(there may be a better way if anyone else knows)
a
alex
10/19/2022, 9:22 PM@Sander Can you verify that
prefect-awss
Sander
10/19/2022, 9:23 PMYeah. I was thinking that could be the issue.
This is the error log generated by the kubernetesjob pod.
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 153, in load_flow_from_flow_run
storage_block = Block._from_block_document(storage_document)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 540, in _from_block_document
else cls.get_block_class_from_schema(block_document.block_schema)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 571, in get_block_class_from_schema
return lookup_type(cls, block_schema_to_key(schema))
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/dispatch.py", line 186, in lookup_type
raise KeyError(
KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."
Hard to copy paste nicely on my phone:)
Trying to find the kubernetesjob manifest
I only see the prefect-job container there.
I think I need to manually install as a customisation. How should I go about that?
a
Andrew Huang
10/19/2022, 9:37 PMcan you elaborate what you mean by customisation? is this a Kubernetes term?
s
Sander
10/19/2022, 9:42 PMWhat I think is happening is that the latest prefect_aws lib is not in the prefect-job container that is used by default in the kubernetes job manifest.
I see that I can customise the manifest in the kubernetesjob block.
However , at the moment I’m not sure how to make sure I update the appropriate libs (pip install prefect_aws in the prefect-job container when it is started as part of the kubernetesjob
a
Andrew Huang
10/19/2022, 9:49 PMs
Sander
10/19/2022, 10:04 PMWill check it out and get back. Thanks!
I think that will help indeed. Will try out EXTRA_PIP_PACKAGES
a
Andrew Huang
10/19/2022, 10:22 PMI’m glad! thanks for checking it out
s
Sander
10/20/2022, 8:44 PMbotocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.I think I need to figure out a bit about my minio setup
Ok getting somewhere.
botocore.exceptions.SSLError: SSL validation failed for <https://minio:443/k8s?prefix=&encoding-type=url> [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)@Andrew Huang would you happen to know if and how I can disable SSL verification when connecting to my local minio tenant running in my local kubernetes cluster?
a
Andrew Huang
10/20/2022, 9:29 PMIs SSLVerification set on boto3 config? if so, this might be related https://github.com/PrefectHQ/prefect-aws/issues/124
s
Sander
10/20/2022, 9:31 PMI’m inclined to think that I need to do some plumbing on my cluster but I’ll definitely read that link. (A bit long to get the gist now :) ) thanks!
👀 1
Having read the link I’m not sure it discusses the same issue. I’ll have to investigate how the SSL auth works on the minio tenant inside my k8s first I guess.
a
Andrew Huang
10/20/2022, 9:52 PMs3 = boto3.client('s3', verify=False)it might be related because there’s no way to pass verify=False atm
s
Sander
10/20/2022, 9:53 PMOh. Btw. The extra pip packages did the job! Thanks
👍 1
a
Andrew Huang
10/20/2022, 9:53 PMcc @alex
s
Sander
10/20/2022, 9:54 PMAh in that sense. No indeed. Then it’s likely related. Nevertheless, I think the kubernetes cluster can auto issue certificates
But not sure.
Now I see. Passing these options to the s3 bucket should definitely solve this SSL issue.
Hi @alex and @Andrew Huang I tried to look at changing the code myself, but I think I’m missing some background knowledge to properly help out here. How can we move this missing feature forward as currently I would need to jump quite some hoops to set up ssl within my cluster and I rather be able to not verify. Is there a solution thinkable where we keep the property issue discussed in the links for later and for now provide a means to avoid the ssl verification? What are your thoughts? Thanks!
👀 1
Let me double check if all was set correctly on my minio tenant.
Just double checked and I also get an ssl error when I try to access the tenant via plain python code so I’m not sure if the feature request will solve my issue.
I’ll get back to you once I’m sure this feature is a real blocker for me.
a
Andrew Huang
10/24/2022, 5:09 PMwhen using plain python code, if you disable verify, are you able to get it working?
s
Sander
10/24/2022, 5:34 PMWas trying that route indeed, but then I needed to create a SSL-less tenant which then clashed with some ports. So need to work more to better understand the setup. What do surprises me though is that I don’t get any complaints from prefect server when I create a manifest with my minio storage block and apply the manifest. Also need to look into that so it’s definitely not a clear cut thing yet.
👀 1
Seems I’m getting somewhere but nothing to report yet. Once I have something I’ll let you know.
👍 1
I have been able to write to the minio in the cluster by disabling SSL via an http_client where I put the cert reqs to CERT_NONE.
Current line of approach is either override SSL via env variables or adjust my minio setup.
My hope is that I can set env variables.
I was indeed able to inject the public certificate that is self signed by the minio tenant into the agent pod. This solves the SSL issue. You can use the secret that is generated by the minio operator and save the public.crt string in the secret onto the pod in the standard SSL directory.
✅ 1