https://prefect.io logo
Title
r

redsquare

10/24/2022, 5:22 PM
Hi All, just upgraded to an Org account and reconfiguring K8's to use the new service accounts - I cant get my k8s agent to start, the agent looks to be connected fine as it writes out the api url but then 404's on getting the {url}/work_queues
Agent started! Looking for work from queue(s): default-work-queue...Mon, Oct 24 2022 6:17:45 pm17:17:45.536 | ERROR | prefect.agent - Failed to create work queue 'default-work-queue'.Mon, Oct 24 2022 6:17:45 pm
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/api/accounts/9396b5ff-23c0-42b4-91d9-5892450xxxx/workspaces/de803906-c6f7-45%0A77-a3aa-d300b56fxxx/work_queues/>'
๐Ÿ‘€ 1
โœ… 1
Ignore me - on copy paste rancher secrets included a line break in the PREFECT_API_URL
โœ… 2
๐Ÿ‘ 1
g

George Coyne

10/24/2022, 6:01 PM
Glad to hear it got sorted out!
๐Ÿ™Œ 1
r

redsquare

10/24/2022, 7:25 PM
Perhaps a better test and console message is needed on connection to see if the url+key is valid rather than 404'ing on the workqueue
c

Christopher Boyd

10/24/2022, 8:55 PM
a little late but I noticed in your url, you have %0A which is the url encoded version of a line feed ``````
de803906-c6f7-45%0A77-a3aa-d300b56fxxx/work_queues/'
r

redsquare

10/24/2022, 8:58 PM
Hey @Christopher Boyd yeah, copy paste issue into rancher from local console ๐Ÿ˜
๐Ÿ™Œ 1
c

Christopher Boyd

10/24/2022, 9:00 PM
I think useful to see the outcome and fix even if itโ€™s simple, I doubt this will be the last time we see this happen ๐Ÿ™‚
r

redsquare

10/24/2022, 9:03 PM
Could a test be done in the agent on connect, currently the logging appears as if it has connected successfully
c

Christopher Boyd

10/24/2022, 9:05 PM
Let me check on a few things, Ill get back shortly
so If I start an agent fresh, I get the following:
Starting v2.6.1 agent connected to 
<https://api.prefect.cloud/api/accounts/><redacted>/workspaces/<redacted>...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): kubernetes...
Intentionally breaking my api url (just a gibbered tenant / workspace): I get a traceback:
Starting v2.6.1 agent connected to 
<https://api.prefect.cloud/api/accounts/><>/work
spaces/<>...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): kubernetes...
21:07:44.022 | ERROR   | prefect.agent - Failed to create work queue 'kubernetes'.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 111, in get_work_queues
    work_queue = await self.client.create_work_queue(name=name)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/orion.py", line 648, in create_work_queue
    response = await <http://self._client.post|self._client.post>("/work_queues/", json=data)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
The client is more or less just reaching the endpoint directly, it doesnโ€™t know if itโ€™s a good or bad URL though since every tenant / workspace will be a different url
I can see what the team thinks about this line though, since I can see it being misleading
Agent started! Looking for work from queue(s): kubernetes...