https://prefect.io logo
Join Slack
Powered by
UPDATE (RESOLVED): Turns out it's an authenticatio...
# ask-community
c
UPDATE (RESOLVED): Turns out it's an authentication 401 error not a 404 error as stated by 
PrefectHTTPStatusError
. I found out that a team member recently changed a few of our k8s secrets and replaced our k8s secret PREFECT_API_KEY with an expired key. Changing the api key to an active key solved the issue. Going to open a GitHub issue as I believe the 404 error sent me on the wrong goose chase... Hello there, deployed Prefect Orion agent to kubernetes connected to Prefect Cloud. Flow runs were working yesterday but today the k8s pods (agent container) started failing with this mysterious error. I cannot seem to find anything wrong with the manifest (it's exactly the same as what we had yesterday except 
image
is now 
prefecthq/prefect:2.2.0-python3.9
instead of 
prefecthq/prefect:2.1.1-python3.9
. Nevertheless, I also tested using image: 
prefecthq/prefect:2.1.1-python3.9
, also getting the same error (see full traceback in thread).
Copy code
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/api/accounts/REDACTED/workspaces/REDACTED/block_types/>'
Response: {'detail': 'Not Found'}
For more information check: <https://httpstatuses.com/404>
An exception occurred.
👍 1
🙏 1
✅ 1
This is our Prefect orion k8s manifest
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orion
spec:
  selector:
    matchLabels:
      app: orion
  replicas: 1
  template:
    metadata:
      labels:
        app: orion
    spec:
      containers:
      - name: api
        image: prefecthq/prefect:2.2.0-python3.9
        command:
        - prefect
        - orion
        - start
        - --host
        - 0.0.0.0
        - --log-level
        - WARNING
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 4200
      - name: agent
        image: prefecthq/prefect:2.2.0-python3.9
        command:
        - prefect
        - agent
        - start
        - -q
        - etl
        imagePullPolicy: IfNotPresent
        env:
        - name: PREFECT_API_URL
          valueFrom:
            secretKeyRef:
              name: prefect-auth
              key: url
        - name: PREFECT_API_KEY
          valueFrom:
            secretKeyRef:
              name: prefect-auth
              key: password
---
apiVersion: v1
kind: Service
metadata:
  name: orion
  labels:
    app: orion
spec:
  ports:
  - port: 4200
    protocol: TCP
  selector:
    app: orion
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  namespace: prefect
  name: flow-runner
rules:
- apiGroups:
  - ''
  resources:
  - pods
  - pods/log
  - pods/status
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: flow-runner-role-binding
  namespace: prefect
subjects:
- kind: ServiceAccount
  name: default
  namespace: prefect
roleRef:
  kind: Role
  name: flow-runner
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
The only changes we made to the manifest were: 1. Added 
PREFECT_API_URL
and 
PREFECT_API_KEY
as env vars in the 
agent
container 2. Changed 
-q kubernetes
to 
-q etl
Full error traceback:
Copy code
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.9/site-packages/prefect/cli/agent.py", line 113, in start
    async with OrionAgent(work_queues=work_queues) as agent:
  File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 282, in __aenter__
    await self.start()
  File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 268, in start
    await self.default_infrastructure._save(is_anonymous=True)
  File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 688, in _save
    await self.register_type_and_schema(client=client)
  File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 637, in register_type_and_schema
    block_type = await client.create_block_type(
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 1013, in create_block_type
    response = await <http://self._client.post|self._client.post>(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 280, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 226, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/api/accounts/REDACTED/workspaces/REDACTED/block_types/>'
Response: {'detail': 'Not Found'}
For more information check: <https://httpstatuses.com/404>
An exception occurred.
k
Just ran into this problem myself with a freshly created key - just wanted to confirm that the 
PREFECT_API_URL
you used was:
Copy code
<https://api.prefect.cloud/api/accounts/REDACTED/workspaces/REDACTED>
?
11 Views
Open in Slack
PreviousNext
Powered by