Can anyone point me to some detailed notes on setting up prefect server and agent on k8s? I have managed to deploy the UI and the agent all pods running green - however the agent does not know where it needs to query the webserver - how do you set the location of the webserver in the agent?

Hi @Josh Greenhalgh! Hmm the agent should be communicating with the API via the cluster.

I have actually worked it out now! Just a timing thing in regards to when the ip is available and not knowing how to update the agents env vars once its running. I solved by deploying the server first waiting for ip then manually creating the agent yml via

prefect agent kubernetes install ...

and passing it the api url once I knew it - I am still unclear as to why the agent is speaking to the server over the open internet rather than within the cluster?

You shouldn’t have to do that, the helm chart configures it to use the same API url (using cluster DNS) that all the other services talk to.

hmmm

It's marked as experimental because we're not committed to backwards compatibility as we develop it since it's so new. Lots of people are using it though and it likely won't remain experimental for long.

So when I start with

--set agent.enabled=true

the agent has the following in its logs;

[2021-01-22 20:02:51,449] ERROR - agent | 400 Client Error: Bad Request for url: <http://prefect-apollo.prefect:4200/graphql>

The following error messages were provided by the GraphQL server:

    INTERNAL_SERVER_ERROR: Variable "$input" got invalid value null at
        "input.tenant_id"; Expected non-nullable type UUID! not to be null.

The GraphQL query was:

    mutation($input: get_runs_in_queue_input!) {
            get_runs_in_queue(input: $input) {
                flow_run_ids
        }
    }

The passed variables were:

    {"input": {"before": "2021-01-22T20:02:51.383799+00:00", "labels": [], "tenant_id": null}}

That’s failing because the default tenant hasn’t been created yet.

Yeah I have done that!

When the agent pod restarts it should be fine then

Ok so yeah you are correct the agent has no logs since then - however the dashboard does not see an agent

Can you try killing the pod to see if a new one works?

kubectl delete pod prefect-agent-59db54f4dc-4lwqb

should remove it and the deployment should spawn a new one

Yep bingo! Is there a way I can turn on much more verbose logging to catch this stuff?

I don’t think I pass through any logging options right now, but in the agent env you could probably do

PREFECT__CLOUD__AGENT__LEVEL=debug

Thanks very much!