https://prefect.io logo
Powered by
Title
k

Keith

08/04/2022, 6:54 AM
Hi all, trying to migrate my Prefect 1 GKE agent to Prefect 2 Orion and have it connect to Prefect Cloud. Followed the excellent demo on YouTube but when the pod starts up I get the following stack trace. I have tried 2.0.1 and 2.0.2 with different Python versions and I get the same error message back each time. Have also tried different ways to configure the kubectl secrets to see if that is the issue, wondering if others have run into the same issue? 
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.10/site-packages/prefect/cli/agent.py", line 93, in start
    async with OrionAgent(
  File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 249, in __aenter__
    await self.start()
  File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 237, in start
    await self.default_infrastructure._save(is_anonymous=True)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 618, in _save
    await self.register_type_and_schema(client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 560, in register_type_and_schema
    block_type = await client.read_block_type_by_slug(
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1093, in read_block_type_by_slug
    return BlockType.parse_obj(response.json())
  File "/usr/local/lib/python3.10/site-packages/httpx/_models.py", line 743, in json
    return jsonlib.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
✅ 1
m

Mason Menges

08/04/2022, 3:51 PM
Hey @Keith can you move the code block into the thread? I'm not particularly familiar with Kubernetes but there is an accompanying discourse article related to the video I think you're referring to that might offer some insight https://discourse.prefect.io/t/prefect-1-0-prefect-2-0-kubernetes-run-config-to-kubernetesjob-infrastructure-block/1229
c

Christopher Boyd

08/04/2022, 3:56 PM
Hi Keith, seems like the error itself is a json decoder error, where it’s expecting json, and not getting that. 
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Where does this error come up specifically ? when you run the kubectl apply? if so, what does your manifest look like?
k

Keith

08/04/2022, 4:32 PM
@Mason Menges thank you, I watched and followed this video exactly and after running 
kubectl apply ...
the pod tries to start up and read the Prefect Cloud API secret setup with this command `kubectl create secret generic api-key --from-literal='prefect_cloud_api_key=<Prefect Cloud API Key>' -n prefect2`:

https://www.youtube.com/watch?v=QBLcLXulR9w▾

@Christopher Boyd Yes, after the 
kubectl apply
I suspect it is trying to read the secret as a JSON but it fails at doing so. I have tried many different combinations of defining the kubectl secret and always get the same JSON issue on deployment of the pod.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-2-agent
  namespace: prefect2
spec:
  selector:
    matchLabels:
      app: prefect-2
  replicas: 1  # We're using SQLite, so we should only run 1 pod
  template:
    metadata:
      labels:
        app: prefect-2
    spec:
      containers:
      - name: api
        image: prefecthq/prefect:2.0.2-python3.10
        command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 4200
      - name: agent
        image: prefecthq/prefect:2.0.2-python3.10
        command: ["prefect", "agent", "start", "kubernetes"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: <https://app.prefect.cloud/account/><Account ID>/workspace/<Workspace ID>
          - name: PREFECT_API_KEY
            valueFrom:
              secretKeyRef:
                name: api-key
                key: prefect_cloud_api_key

---
apiVersion: v1
kind: Service
metadata:
  name: prefect-2-agent
  namespace: prefect2
  labels:
    app: prefect-2
spec:
  ports:
    - port: 4200
      protocol: TCP
  selector:
    app: prefect-2
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  namespace: prefect2
  name: flow-runner
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: flow-runner-role-binding
  namespace: prefect2
subjects:
- kind: ServiceAccount
  name: default
  namespace: prefect2
roleRef:
  kind: Role
  name: flow-runner
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
c

Christopher Boyd

08/04/2022, 4:36 PM
Sure thing - so based on what I see visible in your manifest. the key should be defined as:
- name: PREFECT_API_KEY
            valueFrom:
              secretKeyRef:
                name: api-key
                key: prefect_cloud_api_key
kubectl create secret generic api-key --from-literal=‘prefect_cloud_api_key=<your raw api key here>’
k

Keith

08/04/2022, 4:37 PM
kubectl create secret generic api-key --from-literal='prefect_cloud_api_key=<Prefect Cloud API Key>' -n prefect2
c

Christopher Boyd

08/04/2022, 4:38 PM
just to rule out potential syntax or copy/paste changing dashes and ’, I would recommend trying to type it out explicitly;
yep that looks right
k

Keith

08/04/2022, 4:41 PM
Going to type it out explicitly right now, trying to clear out my old tests
c

Christopher Boyd

08/04/2022, 4:41 PM
I’m copying out your manifest now as well, and testing
k

Keith

08/04/2022, 4:47 PM
c

Christopher Boyd

08/04/2022, 4:48 PM
This was using your manifest you just posted, and creating the key fresh as you see; the only thing that is off screen is my API key, but is something along the lines of: abc_abalkhet1235fsjlk2123
Ok - the only other options I can think of at the immediate moment are:
k

Keith

08/04/2022, 4:48 PM
Basically the pod keeps trying to connect to remote and gets that JSON error
c

Christopher Boyd

08/04/2022, 4:49 PM
Is this your actual URL:? 
<https://app.prefect.cloud/account/><Account ID>/workspace/<Workspace ID>
or are you replacing that?
k

Keith

08/04/2022, 4:50 PM
replacing it
Is it safe to post the whole thing?
c

Christopher Boyd

08/04/2022, 4:50 PM
when you switch to the agent ( I think those logs originage from the API container)
I don’t think you need to, you can withhold it
as long as it’s a valid workspace url - for context, my API logs look like this: 
___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
| _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
|  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
|_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>

View the API reference documentation at <http://0.0.0.0:4200/docs>

Check out the dashboard at <http://0.0.0.0:4200>



INFO:     Started server process [8]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on <http://0.0.0.0:4200> (Press CTRL+C to quit)
My agent log looks like this: 
Starting agent connected to 
<https://api.prefect.cloud/api/accounts/><replaced>/work
spaces/<replaced>...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue 'kubernetes'...
If that’s still not working for you, I would see if you could try a virtual environment with python 3.9? I can’t see anything else that stands out, and I used the manifest you provided to deploy this, so I don’t suspect any syntax errors
actually
one thing
k

Keith

08/04/2022, 4:54 PM
It seems like the Orion server is up but the Agent continually retries
c

Christopher Boyd

08/04/2022, 4:54 PM
You have: 
<https://app.prefect.cloud>
👀 1
My agent is : 
<https://api.prefect.cloud>
k

Keith

08/04/2022, 4:54 PM
I did try a different version and python, tried 
2.0.2-python3.9
& 
2.0.1-python3.9
c

Christopher Boyd

08/04/2022, 4:55 PM
I’d double check the API URL
k

Keith

08/04/2022, 4:55 PM
Good eye, retrying
c

Christopher Boyd

08/04/2022, 4:59 PM
Yep - so
I kicked mine over to app: 
File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
An exception occurred.
I think it’s trying to pull that web-page for your UI login which is html or whatever (not really a front end so I don’t know exactly) instead of the API which is returning a json payload
k

Keith

08/04/2022, 4:59 PM
That worked! I think I missed what was copied at 1:50 timestamp in the YouTube video
c

Christopher Boyd

08/04/2022, 5:00 PM
so it’s expecting a json payload at API but not finding it at APP
👍 1
but that’s a good call out - thank you for letting me know - what I can correct in that video / doc to make it better?
I just tested locally ; I have an idea. Without knowing what anyones API url might be, logging in returns the : “Use this workspace: 
prefect cloud workspace set --workspace "<workspace>"
When executed locally, that sets your API url, and can be displayed : 
prefect config view
🙌 1
k

Keith

08/04/2022, 5:07 PM
This just helped me troubleshoot a follow up issue, definitely a good thing to know
what I can correct in that video
In the video I saw this: 
env:
          - name: PREFECT_API_URL
            value: <https://app.prefect.cloud/account/><>/workspace/<>
          - name: PREFECT_API_KEY
            valueFrom:
              secretKeyRef:
                name: api-key
                key: prefect_cloud_api_key
And missed that there was a API URL at the bottom, which is of the form 
<https://api.prefect.cloud/api/account/><>/workspace/<>
, I think I was just working on typing that template as you were talking and thats how I ended up with the API URL the way it is. I like your suggestion about the 
prefect config view
👍 1
@Christopher Boyd thank you so much for the assistance! Now to test deployments!
🙌 1
6 Views
#prefect-communityJoin Slack
Powered by