Hi I hope you are doing well is there anyone have successful Prefect Community #ask-community

Hi I hope you are doing well, is there anyone hav...

M Taufik

11/13/2020, 4:03 AM

Hi I hope you are doing well, is there anyone have successful a build prefect server in k8s cluster also with k8s agent? could you help share link/github to deploy in k8s cluster with production ready? Thank you

👀 1

Joël Luijmes

11/13/2020, 7:09 AM

Yesterday I deployed prefect + agent: • prefect with helm chart: https://github.com/PrefectHQ/server/tree/master/helm/prefect-server • Agent from documentation: https://docs.prefect.io/orchestration/agents/kubernetes.html This is the command I used to generate the agent yaml:

Copy code

prefect agent kubernetes install --rbac --namespace prefect --api prefect-server-apollo:4200 > prefect-agent.yaml

M Taufik

11/13/2020, 7:39 AM

Hi thank you for your response how about the status of k8s agent able to connect with k8s cluster and running the flow/jobs properly @Joël Luijmes ?

M Taufik

11/13/2020, 7:40 AM

which pods/container to execute that command?

Joël Luijmes

11/13/2020, 7:47 AM

Copy code

prefect agent kubernetes install --rbac --namespace prefect --api prefect-server-apollo:4200 > prefect-agent.yaml

This command generates the required yaml for deployin the agent in kubernetes. (The --api flag points to the apollo server (in cluster).) After generating the yaml, just apply it

kubectl apply -f prefect-agent.yaml

As the helm chart currently don’t have any ingresses, I port forward the apollo and ui services to reach it

M Taufik

11/13/2020, 8:25 AM

ok noted, and how you can create a tenant?

Joël Luijmes

11/13/2020, 8:26 AM

Just followed the instructions in the interfaace Although, I’m unsure what tenant exactly entails (just asked this in the channel).

M Taufik

11/13/2020, 8:29 AM

which container/pods to run command

prefect server create-tenant

Joël Luijmes

11/13/2020, 8:44 AM

just ran the command shown in the command shown in the interface from my localmachine (after portforwarding). But not sure what is the preferred way

M Taufik

11/13/2020, 9:05 AM

i'm facing an error when deploy k8s agent

Copy code

[2020-11-13 09:02:29,526] INFO - agent | Starting KubernetesAgent with labels []
[2020-11-13 09:02:29,526] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-11-13 09:02:29,526] INFO - agent | Agent connecting to the Prefect API at prefect-apollo:4200
[2020-11-13 09:02:29,527] ERROR - agent | There was an error connecting to prefect-apollo:4200
[2020-11-13 09:02:29,527] ERROR - agent | No connection adapters were found for 'prefect-apollo:4200'
[2020-11-13 09:02:29,527] INFO - agent | Waiting for flow runs...
[2020-11-13 09:02:29,530] ERROR - agent | No connection adapters were found for 'prefect-apollo:4200'

Joël Luijmes

11/13/2020, 9:18 AM

The name of

prefect-apollo

kinda depends on how you deployed the helm chart. Check your services with

kubectl get services

and then use the correct name

Joël Luijmes

11/13/2020, 9:19 AM

Oh wait, my example command was wrong, you should include http://

Joël Luijmes

11/13/2020, 9:19 AM

Copy code

prefect agent kubernetes install --rbac --namespace prefect --api <http://prefect-server-apollo:4200> > prefect-agent.yaml

M Taufik

11/13/2020, 9:19 AM

I changed to cluster-ip but still not work

Copy code

[2020-11-13 09:17:25,847] WARNING - agent | Unable to register agent to <http://34.xx.xx.xx:4200>. Make sure the server is running on the latest version.

M Taufik

11/13/2020, 9:20 AM

Copy code

[2020-11-13 09:17:25,848] INFO - agent | Agent connecting to the Prefect API at <http://34.xx.xx.xx:4200>
[2020-11-13 09:17:25,859] INFO - agent | Waiting for flow runs...
[2020-11-13 09:17:26,025] ERROR - agent | 400 Client Error: Bad Request for url: <http://34.xx.xx.xx:4200>

This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
    mutation($input: get_runs_in_queue_input!) {
            get_runs_in_queue(input: $input) {
                flow_run_ids
        }
    }
}
variables {
    {"input": {"before": "2020-11-13T09:17:25.861645+00:00", "labels": [], "tenant_id": null}}

M Taufik

11/13/2020, 9:22 AM

service

Joël Luijmes

11/13/2020, 9:23 AM

I think you should set it to

<http://prefect-apollo:4200>

Joël Luijmes

11/13/2020, 9:24 AM

At least, that works for me, also check you are deploying prefect and the agent in the same namespace

M Taufik

11/13/2020, 9:25 AM

yes, already deploy in same namespace

M Taufik

11/13/2020, 9:25 AM

Copy code

2020-11-13 09:24:27,121] INFO - agent | Starting KubernetesAgent with labels []
[2020-11-13 09:24:27,121] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-11-13 09:24:27,121] INFO - agent | Agent connecting to the Prefect API at <http://prefect-apollo:4200>
[2020-11-13 09:24:27,132] INFO - agent | Waiting for flow runs...
[2020-11-13 09:24:27,236] ERROR - agent | 400 Client Error: Bad Request for url: <http://prefect-apollo:4200>


This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
    mutation($input: get_runs_in_queue_input!) {
            get_runs_in_queue(input: $input) {
                flow_run_ids
        }
    }
}
variables {
    {"input": {"before": "2020-11-13T09:24:27.134876+00:00", "labels": [], "tenant_id": null}}

Joël Luijmes

11/13/2020, 9:28 AM

Ah I see, its connecting. But, honestly i have no idea whats wrong now. I’m new to prefect, maybe someone else has a clue.

M Taufik

11/13/2020, 9:29 AM

seems caused i don't have a tenant,

M Taufik

11/13/2020, 9:30 AM

so to create tenant, are you using prefect server installed in your local server? or running in k8s pods?

M Taufik

11/13/2020, 9:33 AM

i'm trying connect to the k8s pods and run the command facing a connection refused

Joël Luijmes

11/13/2020, 11:11 AM

Ah,I created the tenant from my local machine too

Joël Luijmes

11/13/2020, 11:11 AM

But again, I don’t know the concept of tenant, so not sure what’s best here

M Taufik

11/13/2020, 11:22 AM

so, your local machine is running prefect server, right?

Joël Luijmes

11/13/2020, 11:24 AM

No I dpeloyed to kubernetes, port forward apollo and the ui, then ran the created the tenant from my local machine

Joël Luijmes

11/13/2020, 11:24 AM

And then deployed the agent, also in kubernetes

M Taufik

11/13/2020, 11:27 AM

the port-forward command you running is

kubectl port-forward service/apollo 4200

Joël Luijmes

11/13/2020, 11:29 AM

Almost, for me its

kubectl port-forward svc/prefect-server-apollo 4200

, but that depends on the given name for your deployment. Going of your eerlier screenshot it should be

svc/prefect-apollo

M Taufik

11/13/2020, 11:35 AM

yes, already forwarded

Copy code

kubectl port-forward svc/prefect-apollo 4200
Forwarding from 127.0.0.1:4200 -> 4200

M Taufik

11/13/2020, 11:35 AM

but seems no prefect command installed in my local machine

Copy code

prefect server create-tenant --name default --slug default
-bash: prefect: command not found

Joël Luijmes

11/13/2020, 11:38 AM

Ah in order to run such commands, you should have prefect installed

Joël Luijmes

11/13/2020, 11:39 AM

https://docs.prefect.io/core/getting_started/installation.html#installation

Joël Luijmes

11/13/2020, 11:39 AM

After which you can run prefect commands

M Taufik

11/13/2020, 1:32 PM

after run prefect command in local machine, k8s agent can connect to the server and also create new tenant thank you, I very much appreciate your help

Joël Luijmes

11/13/2020, 1:37 PM

Great you got it working 🙂

Zanie

11/13/2020, 5:08 PM

Thanks @Joël Luijmes 🙂

Zanie

11/13/2020, 5:09 PM

Would expanding the Helm docs to include some port-forward commands be helpful?

Zanie

11/13/2020, 5:09 PM

I’m considering adding agent support to the Helm chart, not sure if that’d be useful as well.

Zanie

11/14/2020, 12:30 AM

If anyone would like to trial this or provide feedback as users of the K8s agent: https://github.com/PrefectHQ/server/pull/136

6 Views

Open in Slack

Previous Next