https://prefect.io logo
j

Jake

01/19/2022, 4:42 PM
Hi everyone, I’m new to prefect and I ran into an issue today. I have the kubernetes agent running in a cluster, the logs show that it is registered and waiting for flow runs. However, this agent does not appear on Prefect Cloud. Any reason why this might be?
k

Kevin Kho

01/19/2022, 4:43 PM
Did you spin up server before trying Prefect Cloud?
j

Jake

01/19/2022, 4:44 PM
No, it’s been cloud the whole time
k

Kevin Kho

01/19/2022, 4:45 PM
Did you login or give the agent an API key?
j

Jake

01/19/2022, 4:46 PM
I gave the agent an API key
k

Kevin Kho

01/19/2022, 4:47 PM
That’s pretty weird. Could you try with a local agent and see if it persists?
j

Jake

01/19/2022, 4:48 PM
I was able to get it working on a minikube cluster
but for some reason, it’s not working on our dev cluster…
• I generated a new API key for the new agent • I changed the namespace to work with our dev cluster everything else remained the same
k

Kevin Kho

01/19/2022, 4:52 PM
the only thing i can think of then is that it’s not hitting the Prefect API…but it sounds like it does and I think that would throw an error.
j

Jake

01/19/2022, 4:54 PM
is it okay that this is blank?
Copy code
- name: PREFECT__CLOUD__TENANT_ID
  value: ''
k

Kevin Kho

01/19/2022, 4:54 PM
I think so because the tenant can be inferred from the API key
j

Jake

01/19/2022, 4:56 PM
I’m running:
Copy code
prefect agent kubernetes install -k <api_key> -l 'cf' --rbac --namespace <namespace> >> prefect_agent.yaml
followed by:
Copy code
kubectl apply -f prefect_agent.yaml --namespace=<namespace>
k

Kevin Kho

01/19/2022, 4:57 PM
that looks right to me
j

Jake

01/19/2022, 4:59 PM
Am I missing something?
z

Zanie

01/19/2022, 5:00 PM
Can you share the agent’s startup logs?
j

Jake

01/19/2022, 5:01 PM
Copy code
[2022-01-19 16:38:50,620] INFO - agent | Registering agent...
[2022-01-19 16:38:51,157] INFO - agent | Registration successful!

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2022-01-19 16:38:51,327] INFO - agent | Starting KubernetesAgent with labels ['cf']
[2022-01-19 16:38:51,327] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-01-19 16:38:51,327] INFO - agent | Waiting for flow runs...
z

Zanie

01/19/2022, 5:02 PM
Would it be easy to enable debug level logs?
j

Jake

01/19/2022, 5:02 PM
How can I do that?
z

Zanie

01/19/2022, 5:04 PM
Set
PREFECT__CLOUD__AGENT__LEVEL=DEBUG
j

Jake

01/19/2022, 5:07 PM
sorry, I’m not sure where to set that? Is that in my install yaml?
a

Anna Geller

01/19/2022, 5:08 PM
You can set it in your YAML file in the env variables section. The only reasons I see why your agent may not appear in Cloud UI: • maybe your backend is on Server? you can switch using: “prefect backend cloud” • perhaps your API key expired - can you try to create a new key?
j

Jake

01/19/2022, 5:10 PM
Copy code
[2022-01-19 17:09:09,787] DEBUG - agent | Environment variables: []
[2022-01-19 17:09:09,788] DEBUG - agent | Max polls: None
[2022-01-19 17:09:09,788] DEBUG - agent | Agent address: http://:8080
[2022-01-19 17:09:09,788] DEBUG - agent | Log to Cloud: True
[2022-01-19 17:09:09,788] DEBUG - agent | Prefect backend: cloud
[2022-01-19 17:09:10,188] DEBUG - agent | Loading incluster configuration
[2022-01-19 17:09:10,189] DEBUG - agent | Namespace: mall-data-test
[2022-01-19 17:09:10,189] INFO - agent | Registering agent...
[2022-01-19 17:09:10,696] INFO - agent | Registration successful!
[2022-01-19 17:09:10,696] DEBUG - agent | Assigned agent id: d6628fec-dc59-4190-9f73-b5c2a90fe19b
[2022-01-19 17:09:10,696] DEBUG - agent | Sending test query to API at '<https://api.prefect.io>'...
[2022-01-19 17:09:10,873] DEBUG - agent | Test query successful!

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2022-01-19 17:09:10,873] INFO - agent | Starting KubernetesAgent with labels ['cf']
[2022-01-19 17:09:10,873] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-01-19 17:09:10,873] INFO - agent | Waiting for flow runs...
[2022-01-19 17:09:10,874] DEBUG - agent | Sending agent heartbeat...
[2022-01-19 17:09:10,880] DEBUG - agent | Running thread pool with 8 workers to handle flow deployment
[2022-01-19 17:09:10,881] DEBUG - agent | Agent API server listening on port http://:8080
[2022-01-19 17:09:10,887] DEBUG - agent | Retrieving information of jobs that are currently in the cluster...
[2022-01-19 17:09:10,887] DEBUG - agent | Querying for ready flow runs...
[2022-01-19 17:09:10,898] DEBUG - agent | Heartbeat succesful! Sleeping for 60.0 seconds...
DEBUG:agent:Heartbeat succesful! Sleeping for 60.0 seconds...
[2022-01-19 17:09:11,064] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
[2022-01-19 17:09:11,065] DEBUG - agent | Sleeping flow run poller for 0.5 seconds...
DEBUG:agent:Sleeping flow run poller for 0.5 seconds...
[2022-01-19 17:09:11,565] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
[2022-01-19 17:09:11,764] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
DEBUG:agent:Sleeping flow run poller for 1.0 seconds...
[2022-01-19 17:09:11,764] DEBUG - agent | Sleeping flow run poller for 1.0 seconds...
DEBUG:agent:Querying for ready flow runs...
[2022-01-19 17:09:12,765] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:No ready flow runs found.
[2022-01-19 17:09:12,936] DEBUG - agent | No ready flow runs found.
DEBUG:agent:Sleeping flow run poller for 2.0 seconds...
[2022-01-19 17:09:12,936] DEBUG - agent | Sleeping flow run poller for 2.0 seconds...
[2022-01-19 17:09:14,937] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
[2022-01-19 17:09:15,109] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
...
And @Anna Geller, the key was just made lass than 1 hour ago and it’s set to never expire. Also I never set the backend to sever, and I think the logs show that it’s using cloud too. Thanks for the suggestions though
a

Anna Geller

01/19/2022, 5:15 PM
It looks good! Could it be that you’re looking at a different tenant? can you share the screenshot of what you see in your agent UI page?
j

Jake

01/19/2022, 5:16 PM
the one shown is the one that was working on my local minikube
a

Anna Geller

01/19/2022, 5:20 PM
it looks like it’s working? or do you try to create another agent with the same label? If you’re deploying another Kubernetes agent, it’s best to attach a different label
j

Jake

01/19/2022, 5:21 PM
this one was created two days ago, shouldn’t the new one show up too?
a

Anna Geller

01/19/2022, 5:21 PM
you can also test it by registering and running some flows on this agent - this way you can see whether the right agent picks up the flow
dunno, I would recommend trying with a different label
j

Jake

01/19/2022, 5:22 PM
Okay let me try that! thanks~
it worked! Adding a different set of labels make it appear 🥳 thanks everyone.
🙌 1
5 Views