Hmm. Seems I was looking at the online manual only. Will check this out.

Ah. Seems to be a known issue: https://github.com/PrefectHQ/prefect-aws/issues/67

@Jeff Hale do you know when this PR is expected to be merged?

Running into other stuff now.

Trying to run my flow as a kubernetes job.

I think prefect version in my kubernetes cluster does have the latest aws lib.

I would try re-registering

prefect block register -m prefect_aws

and then recreate the block

This is the error log generated by the kubernetesjob pod.

Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run flow = await load_flow_from_flow_run(flow_run, client=client) File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 153, in load_flow_from_flow_run storage_block = Block._from_block_document(storage_document) File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 540, in _from_block_document else cls.get_block_class_from_schema(block_document.block_schema) File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 571, in get_block_class_from_schema return lookup_type(cls, block_schema_to_key(schema)) File "/usr/local/lib/python3.10/site-packages/prefect/utilities/dispatch.py", line 186, in lookup_type raise KeyError( KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."

Hard to copy paste nicely on my phone:)

Trying to find the kubernetesjob manifest

I only see the prefect-job container there.

I think I need to manually install as a customisation. How should I go about that?

I see that I can customise the manifest in the kubernetesjob block.

However , at the moment I’m not sure how to make sure I update the appropriate libs (pip install prefect_aws in the prefect-job container when it is started as part of the kubernetesjob

I think that will help indeed. Will try out EXTRA_PIP_PACKAGES

I think I need to figure out a bit about my minio setup

Ok getting somewhere.

@Andrew Huang would you happen to know if and how I can disable SSL verification when connecting to my local minio tenant running in my local kubernetes cluster?

Having read the link I’m not sure it discusses the same issue. I’ll have to investigate how the SSL auth works on the minio tenant inside my k8s first I guess.

it might be related because there’s no way to pass verify=False atm

cc @alex

But not sure.

Now I see. Passing these options to the s3 bucket should definitely solve this SSL issue.

Hi @alex and @Andrew Huang I tried to look at changing the code myself, but I think I’m missing some background knowledge to properly help out here. How can we move this missing feature forward as currently I would need to jump quite some hoops to set up ssl within my cluster and I rather be able to not verify. Is there a solution thinkable where we keep the property issue discussed in the links for later and for now provide a means to avoid the ssl verification? What are your thoughts? Thanks!

Let me double check if all was set correctly on my minio tenant.

Just double checked and I also get an ssl error when I try to access the tenant via plain python code so I’m not sure if the feature request will solve my issue.

I’ll get back to you once I’m sure this feature is a real blocker for me.

Seems I’m getting somewhere but nothing to report yet. Once I have something I’ll let you know.

I have been able to write to the minio in the cluster by disabling SSL via an http_client where I put the cert reqs to CERT_NONE.

Current line of approach is either override SSL via env variables or adjust my minio setup.

My hope is that I can set env variables.

I was indeed able to inject the public certificate that is self signed by the minio tenant into the agent pod. This solves the SSL issue. You can use the secret that is generated by the minio operator and save the public.crt string in the secret onto the pod in the standard SSL directory.