Good morning 2 0 1 question here I am trying connect a Kuber

Question

Good morning 2 0 1 question here I am trying connect a Kubernetes OrionAgent to Prefect Cloud and having some trouble

Billy McMonagle · Answer

I have set the `PREFECT API URL` and `PREFECT API KEY` environment variables I am able to produce the exact error locally that I see in my Kubernetes deployment ``` prefect agent start kubernetes api $PREFECT API URL Starting agent connected to <https app prefect cloud account 251efcb0 78a9 4556 8783 32c643f1f3ef workspace 49ff4325 3577 4a6a 9f13 c6cb19f6591e> Traceback most recent call last File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect utilities asyncutils py line 193 in wrapper return run async in new loop async fn args kwargs File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect utilities asyncutils py line 140 in run async in new loop return anyio run partial fn args kwargs File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 asyncio runners py line 44 in run return loop run until complete main File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 asyncio base events py line 616 in run until complete return future result File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages anyio backends asyncio py line 287 in wrapper return await func args File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect cli agent py line 93 in start async with OrionAgent File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect agent py line 249 in aenter await self start File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect agent py line 237 in start await self default infrastructure save is anonymous=True File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect blocks core py line 618 in save await self register type and schema client=client File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect blocks core py line 560 in register type and schema block type = await client read block type by slug File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect client py line 1091 in read block type by slug return BlockType parse obj response json File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages httpx models py line 743 in json return jsonlib loads self text kwargs File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 json init py line 357 in loads return default decoder decode s File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 json decoder py line 337 in decode obj end = self raw decode s idx= w s 0 end File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 json decoder py line 355 in raw decode raise JSONDecodeError Expecting value s err value from None json decoder JSONDecodeError Expecting value line 1 column 1 char 0 ```

Billy McMonagle · Answer

I believe the Agent expects to receive a DefaultInfrastructure value but I don t know how to provide that value Because I haven t provided it it s attempting to do some sort of temporary block lookup which is unable to complete

Taylor Curran · Answer

Have you looked into creating a `Kubernetes Cluster Config` Block <https docs prefect io api ref prefect blocks kubernetes >

Taylor Curran · Answer

You should see some options in the UI when you go to create a block

Taylor Curran · Answer

There is also a `Kubernetes Job` block

Jeff Hale · Answer

Check out the <https discourse prefect io t prefect 1 0 prefect 2 0 kubernetes run config to kubernetesjob infrastructure block 1229|this Discourse topic> too

Billy McMonagle · Answer

Thank you both I will check out these links I am still a bit stuck on the basic concept of blocks and in this case particularly in understanding why they are necessary for my agent to run and communicate with the API

Taylor Curran · Answer

Sure they are one of the more fresh faced features of 2 0 but as our CEO explains <https www prefect io guide blog the global coordination plane NewandImproved|here> you will see blocks underlying much of Prefect 2 0 s functionality gt Building Blocks The Configuration API gt Blocks are a powerful new way to configure and share business logic We like them so much that many other Prefect 2 0 features are built directly on top of their API including flow and result storage notifications deployments and infrastructure configuration We are continuing to write recipes and examples that show the many different ways blocs can be implemented so thank you for bearing with us as we get those out

Billy McMonagle · Answer

I m trying to create a work queue and getting an error ``` prefect work queue create t test test queue Traceback most recent call last File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect utilities asyncutils py line 193 in wrapper return run async in new loop async fn args kwargs File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect utilities asyncutils py line 140 in run async in new loop return anyio run partial fn args kwargs File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 asyncio runners py line 44 in run return loop run until complete main File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 asyncio base events py line 616 in run until complete return future result File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages anyio backends asyncio py line 287 in wrapper return await func args File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect cli work queue py line 38 in create result = await client create work queue File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect client py line 817 in create work queue response = await work queues json=data File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages httpx client py line 1842 in post return await self request File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages httpx client py line 1527 in request return await self send request auth=auth follow redirects=follow redirects File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect client py line 278 in send response raise for status File Users william mcmonagle pyenv versions 3 8 10 lib python3 8 site packages prefect client py line 224 in raise for status raise PrefectHTTPStatusError from httpx error exc from exc cause prefect exceptions PrefectHTTPStatusError Client error 404 Not Found for url For more information check ``` I believe the URL should reference `work queues` rather than `work queues`

Anna Geller · Answer

< Billy McMonagle> thanks a lot will investigate with the team

Anna Geller · Answer

< Billy McMonagle> looks like the work queue is not an issue we ve seen a similar issue with OSS version where it was necessary to set ```prefect config set PREFECT API URL= ``` but in your case what you need is to deploy your agent as deployment in the same K8s namespace that you want your flow runs to be deployed

Anna Geller · Answer

example ```apiVersion apps v1 kind Deployment metadata name agent namespace dev spec selector matchLabels app agent replicas 1 template metadata labels app agent spec containers name agent image prefecthq prefect 2 python3 9 command prefect agent start tag k8s imagePullPolicy IfNotPresent env name PREFECT API URL value name PREFECT API KEY value xxxx valueFrom secretKeyRef name prefect secrets key api key apiVersion kind Role metadata name agent namespace dev rules apiGroups resources pods pods log pods status verbs get watch list apiGroups batch resources jobs verbs get list watch create update patch delete apiVersion kind RoleBinding metadata name agent namespace dev subjects kind ServiceAccount name default namespace dev roleRef kind Role name agent apiGroup ```

Billy McMonagle · Answer

Got it thanks Anna I do have a deployment setup in the right namespace It seems like it can t start up though For completeness sake my deployment template looks like this ```apiVersion apps v1 kind Deployment metadata name Release Name orion namespace Release Namespace labels Chart Name Release Name orion Release Name spec replicas 1 selector matchLabels Release Name orion template metadata labels Chart Name Release Name orion Release Name cloudwatch log group name Release Name spec containers name agent image Values agent image command prefect agent start kubernetes env name PREFECT API KEY valueFrom secretKeyRef name Release Name key prefect api key name PREFECT API URL valueFrom secretKeyRef name Release Name key prefect api url imagePullPolicy Always resources requests cpu Values agent requests cpu memory Values agent requests memory limits cpu Values agent limits cpu memory Values agent limits memory ```

Billy McMonagle · Answer

FWIW I was able to setup the orion api and the agent talking to each other successfully but I d prefer to connect to Cloud

Billy McMonagle · Answer

The error seems to originate in and the is quite interesting The `OrionAgent start` method now takes the arguments `default infrastructure` and `default infrastructure document id` However doesn t seem to expose those arguments so `default infrastructure` is set to a default of `Process ` I think this code is then trying to save the default infrastructure to a block which it can t do because the agent doesn t have any place to store that information ``` Convert the passed default infrastructure to an id if self default infrastructure and not self default infrastructure document id self default infrastructure document id = await self default infrastructure save is anonymous=True ``` Apologies if I m getting some of this wrong I m still learning the concepts and intended usage

Anna Geller · Answer

sorry no helm expert so can t help with that could you start with the simple K8s deployment I shared and see if that works before moving to helm definitely worth taking it more step by step

Billy McMonagle · Answer

Sure slightly smiling face When using your template I get the 404 work queue error posted up thread which I think is a bug If I switch from `command prefect agent start tag k8s ` to `command prefect agent start k8s ` named queue instead of tagged I get the original message I posted up top of the thread

Anna Geller · Answer

are you using the latest version of Prefect

Billy McMonagle · Answer

Yes I used the image `prefecthq prefect 2 python3 9` tried `prefecthq prefect 2 0 1 python3 9` as well but I think they point to the same image

Anna Geller · Answer

I believe switching to the latest version incl in your Docker images should fix the issue also removing your agent and deployment and recreating from scratch is worth trying

Anna Geller · Answer

correct 2 is always the latest Prefect version

Anna Geller · Answer

but to be sure let me try and see if I can reproduce

Anna Geller · Answer

also crosschecking the namespace is worth doing

Billy McMonagle · Answer

I m deploying on a dev cluster we exclusively use the `default` namespace slightly smiling face I ve copied your template and modified only the env vars to point to my workspace and the namespace

Billy McMonagle · Answer

It s possible there is something wrong with my workspace or with my API key Otherwise I think I m doing it correctly

Anna Geller · Answer

perhaps you can recreate your api key to be sure

Anna Geller · Answer

if you check the pods after triggering the run what does it say is the pod running can you see the logs

Billy McMonagle · Answer

Just created and used a new API key I am able to follow the logs the full output is ``` Starting agent connected to <https app prefect cloud account 251efcb0 78a9 4556 8783 32c643f1f3ef workspace> 49ff4325 3577 4a6a 9f13 c6cb19f6591e Traceback most recent call last File usr local lib python3 9 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs File usr local lib python3 9 site packages prefect utilities asyncutils py line 193 in wrapper return run async in new loop async fn args kwargs File usr local lib python3 9 site packages prefect utilities asyncutils py line 140 in run async in new loop return anyio run partial fn args kwargs File usr local lib python3 9 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File usr local lib python3 9 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File usr local lib python3 9 asyncio runners py line 44 in run return loop run until complete main File usr local lib python3 9 asyncio base events py line 647 in run until complete return future result File usr local lib python3 9 site packages anyio backends asyncio py line 287 in wrapper return await func args File usr local lib python3 9 site packages prefect cli agent py line 93 in start async with OrionAgent File usr local lib python3 9 site packages prefect agent py line 249 in aenter await self start File usr local lib python3 9 site packages prefect agent py line 237 in start await self default infrastructure save is anonymous=True File usr local lib python3 9 site packages prefect blocks core py line 618 in save await self register type and schema client=client File usr local lib python3 9 site packages prefect blocks core py line 560 in register type and schema block type = await client read block type by slug File usr local lib python3 9 site packages prefect client py line 1091 in read block type by slug return BlockType parse obj response json File usr local lib python3 9 site packages httpx models py line 743 in json return jsonlib loads self text kwargs File usr local lib python3 9 json init py line 346 in loads return default decoder decode s File usr local lib python3 9 json decoder py line 337 in decode obj end = self raw decode s idx= w s 0 end File usr local lib python3 9 json decoder py line 355 in raw decode raise JSONDecodeError Expecting value s err value from None json decoder JSONDecodeError Expecting value line 1 column 1 char 0 An exception occurred Stream closed EOF for default agent f49f6454 f2v8k agent ```

Billy McMonagle · Answer

The pod immediately dies Just to clarify I m not doing anything at all with flows I m just trying to get the agent up and reading from a queue

Anna Geller · Answer

Interesting are you still doing it this way ```prefect agent start kubernetes api $PREFECT API URL tag k8s``` or rather ```kubectl apply f agentspec yaml``` the latter is how you are supposed to set this up Not sure if I was clear about this Sorry if not

Billy McMonagle · Answer

No worries the latter is what I was doing Let me know if I can provide any more info

Anna Geller · Answer

Billy I basically can t reproduce Something is different between your and my setup and I wonder what it is I was using Kubernetes on Docker Desktop Perhaps you could share a simple repository with the code you were using redact sensitive info and description of all the steps you took in a README and I could try to reproduce

Billy McMonagle · Answer

Thanks Anna I ll do my best I am glad nobody else seems to be having this issue crossed fingers

Anna Geller · Answer

Honestly it s quite likely it s something super small and silly or perhaps a real bug stuck out tongue winking eye that s the main reason it would be great if we can get an example where both you and me can reproduce the same issue

Anna Geller · Answer

And putting this into a repo and README also forces us to take it slowly and investigate more gradually to find out at what step something went wrong

Billy McMonagle · Answer

Well guess what it was something silly

Billy McMonagle · Answer

It was the contents of the API format itself facepalm ``` Starting agent connected to <https api prefect cloud api accounts xxx work> spaces yyy | | | | | | | | | | | || || | | | | || ` | | | | | | | | | | | | | | | | | | | Agent started Looking for work from queue kubernetes ```

Billy McMonagle · Answer

I had tried lt account id gt workspace lt workspace id gt lt account id gt workspace lt workspace id gt lt account id gt workspace lt workspace id gt api Then I looked back at and realized it contained the correct format flushed lt account id gt workspaces lt workspace id gt

Billy McMonagle · Answer

Thank you so much for your patience on this it was definitely a silly one I know y all are still adding lots of docs every day and I m sure this will be included once there are docs on connecting to Cloud Hopefully my mistake is a good opportunity to throw an extra error message in somewhere sweat smile sweat smile sweat smile Appreciate your time and I look forward to asking another silly question soon

Anna Geller · Answer

oh nooo so much time wasted hahaha smile Nice work figuring that out I should have noticed that too cc < terrence> could we add a note somewhere on the docs more prominently that the correct PREFECT API URL for agents deployed to Prefect Cloud is `````` rather than ``````

terrence · Answer

To follow up I fixed these API URLs in the docs and added additional notes where appropriate Thanks for the heads up Updates will be published with our next release if not sooner

Billy McMonagle · Answer

Thanks Terrence