https://prefect.io logo
Powered by
Title
p

Parwez Noori

08/16/2022, 1:45 PM
Hi everyone! I have the following issue, that I cannot seem to set the secret api key correctly inside the agent: I have verified with this: kubectl get secret api-key --namespace prefect2 -o jsonpath='{.data.prefect_api_key}' | base64 --decode This is done in Azure following Cristopher Boyd:

https://www.youtube.com/watch?v=m9FYoOIfWMc&ab_channel=Prefect▾

✅ 1
c

Christopher Boyd

08/16/2022, 2:03 PM
Hi Parwez, it looks like you have the secret set, but not the API url
env:
            - name: PREFECT_API_URL
              value: <http://orion:4200/api>
            - name: PREFECT_API_KEY
              valueFrom:
                secretKeyRef:
                  name: api-key
                  key: prefect_api_key
so it;s trying to use a secret API key, but not pointing to the cloud API url
b

Bianca Hoch

08/16/2022, 2:34 PM
Also @Parwez Noori, could you move the body of this message and the config file code block to this thread and remove it from the main message? It helps us keep the main community page tidy. 😄
p

Parwez Noori

08/16/2022, 6:53 PM
Hi Christopher, Thank you for your recent video! It was really awesome and has contributed to where we currently are with prefect. That was actually the old version. The new version that I have deployed says, as in your video: 
env:
            - name: PREFECT_API_URL
              value: >-
                <https://app.prefect.cloud/account/75f678b0-8ed7-4ae8-848b-e4eecb4d1df6/workspace/f5a40bc8-5409-48aa-a8df-3ad0a1baf84b>
            - name: PREFECT_API_KEY
              valueFrom:
                secretKeyRef:
                  name: api-key
                  key: prefect_api_key
I still get this error from the agent: 
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.9/site-packages/prefect/cli/agent.py", line 93, in start
    async with OrionAgent(
  File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 267, in __aenter__
    await self.start()
  File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 255, in start
    await self.default_infrastructure._save(is_anonymous=True)
  File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 618, in _save
    await self.register_type_and_schema(client=client)
  File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 560, in register_type_and_schema
    block_type = await client.read_block_type_by_slug(
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 1101, in read_block_type_by_slug
    return BlockType.parse_obj(response.json())
  File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 743, in json
    return jsonlib.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Hi @Bianca Hoch I have done that now. Please let me know if anything else 😄
@Christopher Boyd what is the advantages of running prefect with orion and agent in kubernetes instead of running an vm and connect that to the prefect cloud? Would you still need to run an instance of Orion as well, if you run vm and prefect cloud?
c

Christopher Boyd

08/16/2022, 7:41 PM
@Parwez Noori - 
<https://app.prefect.cloud/account/75f678b0-8ed7-4ae8-848b-e4eecb4d1df6/workspace/f5a40bc8-5409-48aa-a8df-3ad0a1baf84b>
Should be https://api
🙏 1
✅ 1
I think that was an oversight I had in my video when doing the copying and pasting and I didn’t get a chance to update it
🙌 1
Sorry about that!
As for context with kubernetes and vm, it’s a much smaller footprint, and adds the resiliency / reliability to quickly modify the deployment
Regarding with Orion, do you mean prefect server vs prefect cloud? If you have Prefect cloud, you just need the default manifest with the API and agent
otherwise, yea, you could deploy the Orion configuration, then you wouldn’t need an API key or API url
p

Parwez Noori

08/16/2022, 8:07 PM
@Christopher Boyd Thank you a lot! It has now been solved and I'm connected. I have been sitting with the issue the whole day.
✅ 1
c

Christopher Boyd

08/16/2022, 8:26 PM
Sorry about that 😞
p

Parwez Noori

08/17/2022, 7:14 AM
No worries, Chris. Thank you a lot for your help! 😄
@Christopher Boyd I also figured out that the error you had occured because of the infrastructure-block definition. When I include the tag "-ib kubernetes-job" I get the same error, however, when not included it gets deployed. I have not investigated the error any further. My question is that, if it gets deployed as a process instead of a kubernetes-job, you would not be able to leverage your kubernetes cluster for splitting the jobs up for multiple pods? Is that correctly understood and is there any other disadvantages for running it as a process?
c

Christopher Boyd

08/17/2022, 12:30 PM
Hi Parwez, that was an oversight on my part and really more of just a small typo. -ib for infrastructure block is used for pre-defined templates and jobs, while -i is just kubernetes-job
The idea being, you could run it the first time with -i, figure out that you are going to need something repeatable for your deployment definition (for other deployments), and create a block out of it. Then that block could be used with -ib instead
#prefect-communityJoin Slack
Powered by