https://prefect.io logo
Join Slack
Powered by
Good morning, 2.0.1 question here: I am trying con...
# prefect-community
b
Good morning, 2.0.1 question here: I am trying connect a Kubernetes OrionAgent to Prefect Cloud, and having some trouble.
āœ… 1
I have set the 
PREFECT_API_URL
and 
PREFECT_API_KEY
environment variables. I am able to produce the exact error locally that I see in my Kubernetes deployment:
Copy code
āÆ prefect agent start kubernetes --api $PREFECT_API_URL
Starting agent connected to <https://app.prefect.cloud/account/251efcb0-78a9-4556-8783-32c643f1f3ef/workspace/49ff4325-3577-4a6a-9f13-c6cb19f6591e>...
Traceback (most recent call last):
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/cli/agent.py", line 93, in start
    async with OrionAgent(
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/agent.py", line 249, in __aenter__
    await self.start()
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/agent.py", line 237, in start
    await self.default_infrastructure._save(is_anonymous=True)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/blocks/core.py", line 618, in _save
    await self.register_type_and_schema(client=client)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/blocks/core.py", line 560, in register_type_and_schema
    block_type = await client.read_block_type_by_slug(
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/client.py", line 1091, in read_block_type_by_slug
    return BlockType.parse_obj(response.json())
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/httpx/_models.py", line 743, in json
    return jsonlib.loads(self.text, **kwargs)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I believe the Agent expects to receive a DefaultInfrastructure value, but I don't know how to provide that value. Because I haven't provided it, it's attempting to do some sort of temporary block lookup which is unable to complete.
t
Have you looked into creating a 
Kubernetes Cluster Config
Block? https://docs.prefect.io/api-ref/prefect/blocks/kubernetes/
You should see some options in the UI when you go to create a block
There is also a 
Kubernetes Job
block.
j
Check out the this Discourse topic, too.
gratitude thank you 1
b
Thank you both. I will check out these links. I am still a bit stuck on the basic concept of blocks, and in this case particularly, in understanding why they are necessary for my agent to run and communicate with the API.
t
Sure, they are one of the more fresh faced features of 2.0, but as our CEO explains here, you will see blocks underlying much of Prefect 2.0's functionality:
--- Building Blocks: The Configuration API ---
Blocks are a powerful new way to configure and share business logic. We like them so much that many other Prefect 2.0 features are built directly on top of their API, including flow and result storage, notifications, deployments, and infrastructure configuration.
We are continuing to write recipes and examples that show the many different ways blocs can be implemented, so thank you for bearing with us as we get those out!
b
I'm trying to create a work queue and getting an error:
Copy code
āÆ prefect work-queue create -t test test_queue
Traceback (most recent call last):
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/cli/work_queue.py", line 38, in create
    result = await client.create_work_queue(
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/client.py", line 817, in create_work_queue
    response = await <http://self._client.post|self._client.post>("/work_queues/", json=data)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/client.py", line 278, in send
    response.raise_for_status()
  File "/Users/william.mcmonagle/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/client.py", line 224, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://app.prefect.cloud/account/xxx/workspace/yyy/work_queues/>'
For more information check: <https://httpstatuses.com/404>
I believe the URL should reference 
work-queues
rather than 
work_queues
https://github.com/PrefectHQ/prefect/blob/main/src/prefect/client.py#L819
a
@Billy McMonagle thanks a lot, will investigate with the team
@Billy McMonagle looks like the work queue is not an issue we've seen a similar issue with OSS version where it was necessary to set
Copy code
prefect config set PREFECT_API_URL="<http://localhost:4200/api>"
but in your case, what you need is to deploy your agent as deployment in the same K8s namespace that you want your flow runs to be deployed
example:
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent
  namespace: dev
spec:
  selector:
    matchLabels:
      app: agent
  replicas: 1
  template:
    metadata:
      labels:
        app: agent
    spec:
      containers:
        - name: agent
          image: prefecthq/prefect:2-python3.9
          command: ["prefect", "agent", "start", "--tag", "k8s"]
          imagePullPolicy: "IfNotPresent"
          env:
            - name: PREFECT_API_URL
              value: "<https://api.prefect.cloud/api/accounts/c5276cbb-62a2-4501-b64a-74d3d900d781/workspaces/aaeffa0e-13fa-460e-a1f9-79b53c05ab36>"
            - name: PREFECT_API_KEY
              value: xxxx
#              valueFrom:
#                secretKeyRef:
#                  name: prefect-secrets
#                  key: api-key
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: agent
  namespace: dev
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: agent
  namespace: dev
subjects:
  - kind: ServiceAccount
    name: default
    namespace: dev
roleRef:
  kind: Role
  name: agent
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
b
Got it, thanks Anna. I do have a deployment setup in the right namespace. It seems like it can't start up though. For completeness' sake, my deployment template looks like this:
Copy code
apiVersion: apps/v1
kind: Deployment

metadata:
  name: {{ .Release.Name }}-orion
  namespace: {{ .Release.Namespace }}
  labels:
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: {{ .Chart.Name }}
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: {{ .Release.Name }}
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: orion
    <http://tags.datadoghq.com/service|tags.datadoghq.com/service>: {{ .Release.Name }}

spec:
  replicas: 1
  selector:
    matchLabels:
      <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: {{ .Release.Name }}
      <http://app.kubernetes.io/component|app.kubernetes.io/component>: orion
  template:
    metadata:
      labels:
        <http://app.kubernetes.io/name|app.kubernetes.io/name>: {{ .Chart.Name }}
        <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: {{ .Release.Name }}
        <http://app.kubernetes.io/component|app.kubernetes.io/component>: orion
        <http://tags.datadoghq.com/service|tags.datadoghq.com/service>: {{ .Release.Name }}
        cloudwatch/log_group_name: {{ .Release.Name }}
    spec:
      containers:
      - name: agent
        image: {{ .Values.agent.image }}
        command: ["prefect", "agent", "start", "kubernetes"]
        env:
          - name: PREFECT_API_KEY
            valueFrom:
              secretKeyRef:
                name: {{ .Release.Name }}
                key: prefect_api_key
          - name: PREFECT_API_URL
            valueFrom:
              secretKeyRef:
                name: {{ .Release.Name }}
                key: prefect_api_url
        imagePullPolicy: Always
        resources:
          requests:
            cpu: {{ .Values.agent.requests.cpu }}
            memory: {{ .Values.agent.requests.memory }}
          limits:
            cpu: {{ .Values.agent.limits.cpu }}
            memory: {{ .Values.agent.limits.memory }}
FWIW, I was able to setup the orion api and the agent talking to each other successfully, but I'd prefer to connect to Cloud.
The error seems to originate in here in agent.py, and the associated commit is quite interesting. The 
OrionAgent.start
method now takes the arguments 
default_infrastructure
and 
default_infrastructure_document_id
. However, the CLI doesn't seem to expose those arguments, so 
default_infrastructure
is set to a default of 
Process()
. I think this code is then trying to save the default infrastructure to a block, which it can't do because the agent doesn't have any place to store that information.
Copy code
# Convert the passed default infrastructure to an id
        if self.default_infrastructure and not self.default_infrastructure_document_id:
            self.default_infrastructure_document_id = (
                await self.default_infrastructure._save(is_anonymous=True)
            )
Apologies if I'm getting some of this wrong, I'm still learning the concepts and intended usage.
a
sorry, no helm expert, so can't help with that - could you start with the simple K8s deployment I shared and see if that works before moving to helm? definitely worth taking it more step by step
b
Sure šŸ™‚. When using your template, I get the 404 work-queue error posted up thread, which I think is a bug. If I switch from 
command: ["prefect", "agent", "start", "--tag", "k8s"]
to 
command: ["prefect", "agent", "start", "k8s"]
(named queue instead of tagged) I get the original message I posted up top of the thread.
a
are you using the latest version of Prefect?
b
Yes, I used the image 
prefecthq/prefect:2-python3.9
(tried 
prefecthq/prefect:2.0.1-python3.9
as well but I think they point to the same image)
a
I believe switching to the latest version, incl. in your Docker images should fix the issue also removing your agent and deployment and recreating from scratch is worth trying
correct 2-* is always the latest Prefect version
šŸ‘ 1
but to be sure, let me try and see if I can reproduce
also crosschecking the namespace is worth doing
b
I'm deploying on a dev cluster, we exclusively use the 
default
namespace šŸ™‚. I've copied your template and modified only the env vars to point to my workspace and the namespace.
It's possible there is something wrong with my workspace or with my API key? Otherwise I think I'm doing it correctly.
a
perhaps you can recreate your api key to be sure?
if you check the pods after triggering the run, what does it say? is the pod running? can you see the logs?
b
Just created and used a new API key. I am able to follow the logs, the full output is...
Copy code
ā”‚ Starting agent connected to                                                                                                                                                                                                                       ā”‚ā”‚ <https://app.prefect.cloud/account/251efcb0-78a9-4556-8783-32c643f1f3ef/workspace>                                                                                                                                                                  ā”‚ā”‚ /49ff4325-3577-4a6a-9f13-c6cb19f6591e...                                                                                                                                                                                                          ā”‚ā”‚ Traceback (most recent call last):                                                                                                                                                                                                                ā”‚ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper                                                                                                                                                    ā”‚ā”‚     return fn(*args, **kwargs)                                                                                                                                                                                                                    ā”‚ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper                                                                                                                                             ā”‚ā”‚     return run_async_in_new_loop(async_fn, *args, **kwargs)                                                                                                                                                                                       ā”‚ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop                                                                                                                               ā”‚ā”‚     return anyio.run(partial(__fn, *args, **kwargs))                                                                                                                                                                                              ā”‚ā”‚   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run                                                                                                                                                        ā”‚ā”‚     return asynclib.run(func, *args, **backend_options)                                                                                                                                                                                           ā”‚ā”‚   File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run                                                                                                                                                     ā”‚ā”‚     return native_run(wrapper(), debug=debug)                                                                                                                                                                                                     ā”‚ā”‚   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run                                                                                                                                                                             ā”‚ā”‚     return loop.run_until_complete(main)                                                                                                                                                                                                          ā”‚ā”‚   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete                                                                                                                                                         ā”‚ā”‚     return future.result()                                                                                                                                                                                                                        ā”‚ā”‚   File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper                                                                                                                                                 ā”‚ā”‚     return await func(*args)                                                                                                                                                                                                                      ā”‚ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/cli/agent.py", line 93, in start                                                                                                                                                           ā”‚ā”‚     async with OrionAgent(                                                                                                                                                                                                                        ā”‚ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 249, in __aenter__                                                                                                                                                         
ā”‚     await self.start()                                                                                                                                                                                                                            
ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 237, in start                                                                                                                                                              
ā”‚     await self.default_infrastructure._save(is_anonymous=True)                                                                                                                                                                                    
ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 618, in _save                                                                                                                                                        
ā”‚     await self.register_type_and_schema(client=client)                                                                                                                                                                                            
ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 560, in register_type_and_schema                                                                                                                                     
ā”‚     block_type = await client.read_block_type_by_slug(                                                                                                                                                                                            
ā”‚   File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 1091, in read_block_type_by_slug                                                                                                                                          
ā”‚     return BlockType.parse_obj(response.json())                                                                                                                                                                                                   
ā”‚   File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 743, in json                                                                                                                                                               
ā”‚     return jsonlib.loads(self.text, **kwargs)                                                                                                                                                                                                     
ā”‚   File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads                                                                                                                                                                            
ā”‚     return _default_decoder.decode(s)                                                                                                                                                                                                             
ā”‚   File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode                                                                                                                                                                            
ā”‚     obj, end = self.raw_decode(s, idx=_w(s, 0).end())                                                                                                                                                                                             
ā”‚   File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode                                                                                                                                                                        
ā”‚     raise JSONDecodeError("Expecting value", s, err.value) from None                                                                                                                                                                              
ā”‚ json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)                                                                                                                                                                           
ā”‚ An exception occurred.                                                                                                                                                                                                                            
ā”‚ Stream closed EOF for default/agent-f49f6454-f2v8k (agent)
The pod immediately dies. Just to clarify, I'm not doing anything at all with flows, I'm just trying to get the agent up and reading from a queue.
a
Interesting - are you still doing it this way?
Copy code
prefect agent start kubernetes --api $PREFECT_API_URL --tag k8s
or rather
Copy code
kubectl apply -f agentspec.yaml
the latter is how you are supposed to set this up. Not sure if I was clear about this. Sorry if not
b
No worries - the latter is what I was doing. Let me know if I can provide any more info.
a
Billy, I basically can't reproduce. Something is different between your and my setup and I wonder what it is. I was using Kubernetes on Docker Desktop. Perhaps you could share a simple repository with the code you were using (redact sensitive info) and description of all the steps you took in a README and I could try to reproduce?
b
Thanks Anna. I'll do my best. I am glad nobody else seems to be having this issue šŸ¤ž
a
Honestly it's quite likely it's something super small and silly or perhaps a real bug šŸ˜œ that's the main reason it would be great if we can get an example where both you and me can reproduce the same issue
And putting this into a repo and README also forces us to take it slowly and investigate more gradually to find out at what step something went wrong
b
Well.... guess what, it was something silly!
It was the contents of the API format itself facepalm
Copy code
ā”‚ Starting agent connected to                                                                                                                                                                                                                       ā”‚
ā”‚ <https://api.prefect.cloud/api/accounts/xxx/work>                                                                                                                                                                  ā”‚
ā”‚ spaces/yyy...                                                                                                                                                                                                    ā”‚
ā”‚                                                                                                                                                                                                                                                   ā”‚
ā”‚   ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____                                                                                                                                                                                        ā”‚
ā”‚  | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|                                                                                                                                                                                       ā”‚
ā”‚  |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |                                                                                                                                                                                         ā”‚
ā”‚  |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|                                                                                                                                                                                         ā”‚
ā”‚                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
ā”‚ Agent started! Looking for work from queue 'kubernetes'...
I had tried: ā€¢ https://app.prefect.cloud/account/<account-id>/workspace/<workspace-id> ā€¢ https://api.prefect.cloud/account/<account-id>/workspace/<workspace-id> ā€¢ https://app.prefect.cloud/account/<account-id>/workspace/<workspace-id>/api Then I looked back at this message and realized it contained the correct format šŸ˜³ : ā€¢ https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>
Thank you so much for your patience on this, it was definitely a silly one. I know y'all are still adding lots of docs every day, and I'm sure this will be included once there are docs on connecting to Cloud. Hopefully my mistake is a good opportunity to throw an extra error message in somewhere šŸ˜… šŸ˜… šŸ˜… . Appreciate your time and I look forward to asking another silly question soon....
šŸ™Œ 2
šŸ™ 2
āœ… 2
a
oh nooo!!! so much time wasted hahaha šŸ˜„ Nice work figuring that out. I should have noticed that too! cc @terrence could we add a note somewhere on the docs more prominently that the correct PREFECT_API_URL for agents deployed to Prefect Cloud is:
Copy code
<https://api.prefect.cloud/api/accounts/uuid/workspaces/uuid>
rather than:
Copy code
<https://app.prefect.cloud/api/accounts/uuid/workspaces/uuid>
šŸ‘ 1
t
To follow up, I fixed these API URLs in the docs and added additional notes where appropriate. Thanks for the heads up. Updates will be published with our next release, if not sooner.
šŸ™ 3
šŸ‘ 2
b
Thanks Terrence!
5 Views
Open in Slack
PreviousNext
Powered by