https://prefect.io logo
Join Slack
Powered by
Hey team. I'm getting an error to deploy the agent...
# ask-community
m
Hey team. I'm getting an error to deploy the agent on Kubernetes. This is what I am running
Copy code
export PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/><ACCOUNT-ID>/workspaces/<WORKSPACE-ID>"
export PREFECT_API_KEY="<API-KEY>"
prefect kubernetes manifest agent -q <MY-QUEUE> | kubectl apply --namespace=default -f -
And the error..
Copy code
| _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): <MY-QUEUE>...
An exception occurred.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 230, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 181, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/agent.py", line 201, in start
    tg.start_soon(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/services.py", line 46, in critical_service_loop
    await workload()
  File "/usr/local/lib/python3.8/site-packages/prefect/agent.py", line 265, in check_for_cancelled_flow_runs
    named_cancelling_flow_runs = await self.client.read_flow_runs(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/orchestration.py", line 1715, in read_flow_runs
    response = await <http://self._client.post|self._client.post>(f"/flow_runs/filter", json=body)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 253, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 130, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/api/accounts/><ACCOUNT-ID>/workspaces/<WORKSPACE-ID>/flow_runs/filter'
Response: {'detail': 'Not Found'}
For more information check: <https://httpstatuses.com/404>
Why do I get 
Not Found
on 
<https://api.prefect.cloud/api/accounts/><ACCOUNT-ID>/workspaces/<WORKSPACE-ID>/flow_runs/filter
?
z
Hm the 404 seems to hint that you’re not authenticated properly
Could you try to login with 
prefect cloud login
?
m
Hi @Zanie! It shouldn't be authentication as the manifest file applied to the EKS cluster on AWS has the environment variable 
PREFECT_API_KEY
set correctly.
Copy code
- name: agent
          image: prefecthq/prefect:2.8.3-python3.8
          command: ["prefect", "agent", "start", "-q", "<MY-QUEUE>"]
          imagePullPolicy: "IfNotPresent"
          env:
            - name: PREFECT_API_URL
              value: <URL-VALUE>
            - name: PREFECT_API_KEY
              value: <KEY-VALUE>
I tried running a CURL on the URL that is failing in the agent and 
flow_runs/filter
doesn't work on my local either. But a simple command to retrieve the workspace proves that authentication is not an issue. See this: # CURL from my local, to get the workspace (working)
Copy code
curl -X GET -H "Authorization: Bearer $PREFECT_API_KEY" <https://api.prefect.cloud/api/accounts/><ACCOUNT-ID>/workspaces/<WORKSPACE-ID>

{"id":"7e377cd6-45d4-43e5-8999-4788584733ac","created":"2022-09-28T22:57:49.674598+00:00","updated":"2023-02-26T19:48:40.597564+00:00"....}
# CURL from my local, of the command that fails inside the agent
Copy code
curl -X GET -H "Authorization: Bearer $PREFECT_API_KEY" <https://api.prefect.cloud/api/accounts/><ACCOUNT-ID>/workspaces/<WORKSPACE-ID>/flow_runs/filter

{"detail":"Not Found"}
c
Hi Maikel, I'm an engineer for Prefect Cloud, I can help you with this. I'll DM you
For folks who hit this in the future, we have a somewhat subtle restriction: Service accounts added at the organization level do not automatically inherit permissions to all workspaces, only humans do. We do this to help prevent mixups like cross-wiring 
dev
and 
production
workspaces, for example. So the solution here was to grant the SA explicit access to the workspaces that Maikel wanted to run an agent for.
👏 1
s
Hi @Chris Guidry I'm facing the same error ,how do I assign this specific permission to workspaces?Could you please let me know
Hi @Maikel Penz Would like to know inputs from how are you able to solve this ?
m
@sjammula when you create your API key make sure you assign a workspace to it in the dropdown list. This should be the Workspace you are spinning up the agent.
s
Thanks @Maikel Penz for the response.I don't see any workspace in the dropdown .Here is the screenshot what I see when trying to create new apikey in Prefect2
m
try creating a service account
s
sure @Maikel Penz let me do
so basically we need to create an organization then only this service account will be available.correct my understanding @Maikel Penz
m
Instead of creating an api key that is tied to your user you create a service account that has permission to your workspace. You need to be logged into your account/organization yes.
s
thanks @Maikel Penz
c
Hi @sjammula, sorry I was PTO when you sent this reply. What @Maikel Penz has described here is exactly what you'll need to do, create a Service Account and then grant it access to one or more of your organization's workspaces. If your workspaces represent different promotion environments (like 
development
vs 
production
) we recommend using separate service accounts for those.
Thank you so much for fielding this, Maikel!
💪 1
2 Views
Open in Slack
PreviousNext
Powered by