Has anyone seen this error?

File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 292, in __deserialize
    klass = getattr(kubernetes.client.models, klass)
AttributeError: module 'kubernetes.client' has no attribute 'models'

I am running with an out of the box

KubernetesJob

infra and everything was working up until the end of last week. The k8s pods are created and run just fine but Prefect throws this exception while reading the response from the k8s api.

hmm that's a weird error.. that module definitely exists

❯ ipython
Python 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:25:12) [Clang 14.0.6 ]

In [1]: !pip list | rg kubernetes
kubernetes                  26.1.0

In [2]: import kubernetes

In [3]: len(dir(kubernetes.client.models))
Out[3]: 1048

is that error being thrown by an agent running in your cluster?

I am realizing now that I don't believe Prefect is involved here at at all. The error is coming from my own code when creating a k8s pod via a Prefect task. The pod is still created but the api response cannot be deserialized.