Channels
#prefect-community
Title
j
Jake
01/19/2022, 4:42 PMHi everyone, I’m new to prefect and I ran into an issue today. I have the kubernetes agent running in a cluster, the logs show that it is registered and waiting for flow runs. However, this agent does not appear on Prefect Cloud. Any reason why this might be?
k
Kevin Kho
01/19/2022, 4:43 PM
Did you spin up server before trying Prefect Cloud?
j
Jake
01/19/2022, 4:44 PMNo, it’s been cloud the whole time
k
Kevin Kho
01/19/2022, 4:45 PM
Did you login or give the agent an API key?
j
Jake
01/19/2022, 4:46 PMI gave the agent an API key
k
Kevin Kho
01/19/2022, 4:47 PM
That’s pretty weird. Could you try with a local agent and see if it persists?
j
Jake
01/19/2022, 4:48 PMI was able to get it working on a minikube cluster
but for some reason, it’s not working on our dev cluster…
• I generated a new API key for the new agent
• I changed the namespace to work with our dev cluster
everything else remained the same
k
Kevin Kho
01/19/2022, 4:52 PM
the only thing i can think of then is that it’s not hitting the Prefect API…but it sounds like it does and I think that would throw an error.
j
Jake
01/19/2022, 4:54 PMis it okay that this is blank?
Copy code
- name: PREFECT__CLOUD__TENANT_ID
value: ''k
Kevin Kho
01/19/2022, 4:54 PM
I think so because the tenant can be inferred from the API key
j
Jake
01/19/2022, 4:56 PMI’m running:
Copy code
prefect agent kubernetes install -k <api_key> -l 'cf' --rbac --namespace <namespace> >> prefect_agent.yaml Copy code
kubectl apply -f prefect_agent.yaml --namespace=<namespace>k
Kevin Kho
01/19/2022, 4:57 PM
that looks right to me
j
Jake
01/19/2022, 4:59 PMAm I missing something?
z
Zanie
01/19/2022, 5:00 PM
Can you share the agent’s startup logs?
j
Jake
01/19/2022, 5:01 PM Copy code
[2022-01-19 16:38:50,620] INFO - agent | Registering agent...
[2022-01-19 16:38:51,157] INFO - agent | Registration successful!
____ __ _ _ _
| _ \ _ __ ___ / _| ___ ___| |_ / \ __ _ ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __| / _ \ / _` |/ _ \ '_ \| __|
| __/| | | __/ _| __/ (__| |_ / ___ \ (_| | __/ | | | |_
|_| |_| \___|_| \___|\___|\__| /_/ \_\__, |\___|_| |_|\__|
|___/
[2022-01-19 16:38:51,327] INFO - agent | Starting KubernetesAgent with labels ['cf']
[2022-01-19 16:38:51,327] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-01-19 16:38:51,327] INFO - agent | Waiting for flow runs...z
Zanie
01/19/2022, 5:02 PM
Would it be easy to enable debug level logs?
j
Jake
01/19/2022, 5:02 PMHow can I do that?
z
Zanie
01/19/2022, 5:04 PM
Set
PREFECT__CLOUD__AGENT__LEVEL=DEBUGj
Jake
01/19/2022, 5:07 PMsorry, I’m not sure where to set that? Is that in my install yaml?
a
Anna Geller
01/19/2022, 5:08 PM
You can set it in your YAML file in the env variables section.
The only reasons I see why your agent may not appear in Cloud UI:
• maybe your backend is on Server? you can switch using: “prefect backend cloud”
• perhaps your API key expired - can you try to create a new key?
j
Jake
01/19/2022, 5:10 PM Copy code
[2022-01-19 17:09:09,787] DEBUG - agent | Environment variables: []
[2022-01-19 17:09:09,788] DEBUG - agent | Max polls: None
[2022-01-19 17:09:09,788] DEBUG - agent | Agent address: http://:8080
[2022-01-19 17:09:09,788] DEBUG - agent | Log to Cloud: True
[2022-01-19 17:09:09,788] DEBUG - agent | Prefect backend: cloud
[2022-01-19 17:09:10,188] DEBUG - agent | Loading incluster configuration
[2022-01-19 17:09:10,189] DEBUG - agent | Namespace: mall-data-test
[2022-01-19 17:09:10,189] INFO - agent | Registering agent...
[2022-01-19 17:09:10,696] INFO - agent | Registration successful!
[2022-01-19 17:09:10,696] DEBUG - agent | Assigned agent id: d6628fec-dc59-4190-9f73-b5c2a90fe19b
[2022-01-19 17:09:10,696] DEBUG - agent | Sending test query to API at '<https://api.prefect.io>'...
[2022-01-19 17:09:10,873] DEBUG - agent | Test query successful!
____ __ _ _ _
| _ \ _ __ ___ / _| ___ ___| |_ / \ __ _ ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __| / _ \ / _` |/ _ \ '_ \| __|
| __/| | | __/ _| __/ (__| |_ / ___ \ (_| | __/ | | | |_
|_| |_| \___|_| \___|\___|\__| /_/ \_\__, |\___|_| |_|\__|
|___/
[2022-01-19 17:09:10,873] INFO - agent | Starting KubernetesAgent with labels ['cf']
[2022-01-19 17:09:10,873] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-01-19 17:09:10,873] INFO - agent | Waiting for flow runs...
[2022-01-19 17:09:10,874] DEBUG - agent | Sending agent heartbeat...
[2022-01-19 17:09:10,880] DEBUG - agent | Running thread pool with 8 workers to handle flow deployment
[2022-01-19 17:09:10,881] DEBUG - agent | Agent API server listening on port http://:8080
[2022-01-19 17:09:10,887] DEBUG - agent | Retrieving information of jobs that are currently in the cluster...
[2022-01-19 17:09:10,887] DEBUG - agent | Querying for ready flow runs...
[2022-01-19 17:09:10,898] DEBUG - agent | Heartbeat succesful! Sleeping for 60.0 seconds...
DEBUG:agent:Heartbeat succesful! Sleeping for 60.0 seconds...
[2022-01-19 17:09:11,064] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
[2022-01-19 17:09:11,065] DEBUG - agent | Sleeping flow run poller for 0.5 seconds...
DEBUG:agent:Sleeping flow run poller for 0.5 seconds...
[2022-01-19 17:09:11,565] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
[2022-01-19 17:09:11,764] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
DEBUG:agent:Sleeping flow run poller for 1.0 seconds...
[2022-01-19 17:09:11,764] DEBUG - agent | Sleeping flow run poller for 1.0 seconds...
DEBUG:agent:Querying for ready flow runs...
[2022-01-19 17:09:12,765] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:No ready flow runs found.
[2022-01-19 17:09:12,936] DEBUG - agent | No ready flow runs found.
DEBUG:agent:Sleeping flow run poller for 2.0 seconds...
[2022-01-19 17:09:12,936] DEBUG - agent | Sleeping flow run poller for 2.0 seconds...
[2022-01-19 17:09:14,937] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
[2022-01-19 17:09:15,109] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
...And @Anna Geller, the key was just made lass than 1 hour ago and it’s set to never expire. Also I never set the backend to sever, and I think the logs show that it’s using cloud too. Thanks for the suggestions though
a
Anna Geller
01/19/2022, 5:15 PM
It looks good! Could it be that you’re looking at a different tenant? can you share the screenshot of what you see in your agent UI page?
j
Jake
01/19/2022, 5:16 PMthe one shown is the one that was working on my local minikube
a
Anna Geller
01/19/2022, 5:20 PM
it looks like it’s working? or do you try to create another agent with the same label? If you’re deploying another Kubernetes agent, it’s best to attach a different label
j
Jake
01/19/2022, 5:21 PMthis one was created two days ago, shouldn’t the new one show up too?
a
Anna Geller
01/19/2022, 5:21 PM
you can also test it by registering and running some flows on this agent - this way you can see whether the right agent picks up the flow
dunno, I would recommend trying with a different label
j
Jake
01/19/2022, 5:22 PMOkay let me try that! thanks~
it worked! Adding a different set of labels make it appear 🥳 thanks everyone.
🙌 1
5 Views