Hey all am currently running Prefect 2 3 2 on a minikube clu

Question

Hey all am currently running Prefect 2 3 2 on a minikube cluster and while the agent connects to Orion just fine it does not seem to be reading jobs from the queue I have a pending job and the agent just doesn t pick it up Any ideas

José Duarte · Answer

More information I am using an anonymous deployment which I inspected using the CLI and got the following ```Traceback most recent call last File Users duarte Documents data pdbt venv lib python3 8 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs File Users duarte Documents data pdbt venv lib python3 8 site packages prefect utilities asyncutils py line 212 in wrapper return run async in new loop async fn args kwargs File Users duarte Documents data pdbt venv lib python3 8 site packages prefect utilities asyncutils py line 141 in run async in new loop return anyio run partial fn args kwargs File Users duarte Documents data pdbt venv lib python3 8 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File Users duarte Documents data pdbt venv lib python3 8 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File Users duarte pyenv versions 3 8 9 lib python3 8 asyncio runners py line 44 in run return loop run until complete main File Users duarte pyenv versions 3 8 9 lib python3 8 asyncio base events py line 616 in run until complete return future result File Users duarte Documents data pdbt venv lib python3 8 site packages anyio backends asyncio py line 287 in wrapper return await func args File Users duarte Documents data pdbt venv lib python3 8 site packages prefect cli deployment py line 152 in inspect deployment json infrastructure = Block from block document File Users duarte Documents data pdbt venv lib python3 8 site packages prefect blocks core py line 514 in from block document block = block cls parse obj block document data File pydantic main py line 526 in pydantic main BaseModel parse obj File Users duarte Documents data pdbt venv lib python3 8 site packages prefect blocks core py line 171 in init super init args kwargs File pydantic main py line 342 in pydantic main BaseModel init pydantic error wrappers ValidationError 1 validation error for KubernetesJob job value is not a valid dict type=type error dict An exception occurred ``` I am not sure that this is the root cause but I wish Prefect was noisy instead of failing silently

Christopher Boyd · Answer

Hi Jose So you are running minikube locally with an agent pointing to minikube is the agent in the cluster or local Can you interact with the orion instance in any other way e g creating work queues and seeing they exist or blocks

José Duarte · Answer

Hey < Christopher Boyd> everything is running in the minikube cluster I am able to interact with Orion because my Agent s init container registers all flows in the python module as deployments and they re all there

Christopher Boyd · Answer

I think the two main requirements for the agent to pick up the flow then are ensuring that your kubeconfig to talk to the cluster is accurate between the agent + the cluster and ensuring tags work queue are appropriately set Beyond that would need to see how the deployment is registered how the flowrun is created executed and logs from the agent to see what it s thinking

José Duarte · Answer

The work queue I am using is the default one The agent has no logs though is there a verbosity config The deployment may be wrong but Prefect only showed me the error above when I inspected it The flow run is created by clicking on Run in Orion s Deployment

Christopher Boyd · Answer

The agent should definitely have logs by default of what work queue its listening to what workspace its connected to and updates as it is looking for flow runs

José Duarte · Answer

It doesn t do any kind of updates It just says it s listening to the default queue

Christopher Boyd · Answer

did you follow any sort of guide to set this up

José Duarte · Answer

More or less I started by doing everything locally then docker compose to get the containers right and now minikube before moving this to Azure

José Duarte · Answer

I ve gathered the information from several resources as there is no proper guide on self hosting prefect

Christopher Boyd · Answer

There are some resources available for this options 1 and 2 are for self hosted option 3 is to the cloud Additionally this is setup for azure but steps 8 and on should apply across all other environments

José Duarte · Answer

I ve done all those steps though my deployment run just doesn t seem to hit the queue

José Duarte · Answer

< Christopher Boyd> after more investigation I suspect that this had to do with the fact that my `job` field in the `KubernetesJob`was not a dict Prefect is only validating the blocks after the fact not during construction or registering Either way I ve hit another bug and I am currently reporting it