https://prefect.io
Join Slack
Hi I hope you are doing well, is there anyone have successful a build prefect server in k8s cluster...
# ask-community
m
Hi I hope you are doing well, is there anyone have successful a build prefect server in k8s cluster also with k8s agent? could you help share link/github to deploy in k8s cluster with production ready? Thank you
šŸ‘€ 1
j
Yesterday I deployed prefect + agent: ā€¢ prefect with helm chart: https://github.com/PrefectHQ/server/tree/master/helm/prefect-server ā€¢ Agent from documentation: https://docs.prefect.io/orchestration/agents/kubernetes.html This is the command I used to generate the agent yaml:
Copy code
prefect agent kubernetes install --rbac --namespace prefect --api prefect-server-apollo:4200 > prefect-agent.yaml
m
Hi thank you for your response how about the status of k8s agent able to connect with k8s cluster and running the flow/jobs properly @Joƫl Luijmes ?
which pods/container to execute that command?
j
Copy code
prefect agent kubernetes install --rbac --namespace prefect --api prefect-server-apollo:4200 > prefect-agent.yaml
This command generates the required yaml for deployin the agent in kubernetes. (The --api flag points to the apollo server (in cluster).) After generating the yaml, just apply it 
kubectl apply -f prefect-agent.yaml
As the helm chart currently donā€™t have any ingresses, I port forward the apollo and ui services to reach it
m
ok noted, and how you can create a tenant?
j
Just followed the instructions in the interfaace Although, Iā€™m unsure what tenant exactly entails (just asked this in the channel).
m
which container/pods to run command 
prefect server create-tenant
?
j
just ran the command shown in the command shown in the interface from my localmachine (after portforwarding). But not sure what is the preferred way
m
i'm facing an error when deploy k8s agent
Copy code
[2020-11-13 09:02:29,526] INFO - agent | Starting KubernetesAgent with labels []
[2020-11-13 09:02:29,526] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-11-13 09:02:29,526] INFO - agent | Agent connecting to the Prefect API at prefect-apollo:4200
[2020-11-13 09:02:29,527] ERROR - agent | There was an error connecting to prefect-apollo:4200
[2020-11-13 09:02:29,527] ERROR - agent | No connection adapters were found for 'prefect-apollo:4200'
[2020-11-13 09:02:29,527] INFO - agent | Waiting for flow runs...
[2020-11-13 09:02:29,530] ERROR - agent | No connection adapters were found for 'prefect-apollo:4200'
j
The name of 
prefect-apollo
kinda depends on how you deployed the helm chart. Check your services with 
kubectl get services
and then use the correct name
Oh wait, my example command was wrong, you should include http://
Copy code
prefect agent kubernetes install --rbac --namespace prefect --api <http://prefect-server-apollo:4200> > prefect-agent.yaml
m
I changed to cluster-ip but still not work
Copy code
[2020-11-13 09:17:25,847] WARNING - agent | Unable to register agent to <http://34.xx.xx.xx:4200>. Make sure the server is running on the latest version.
Copy code
[2020-11-13 09:17:25,848] INFO - agent | Agent connecting to the Prefect API at <http://34.xx.xx.xx:4200>
[2020-11-13 09:17:25,859] INFO - agent | Waiting for flow runs...
[2020-11-13 09:17:26,025] ERROR - agent | 400 Client Error: Bad Request for url: <http://34.xx.xx.xx:4200>

This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
    mutation($input: get_runs_in_queue_input!) {
            get_runs_in_queue(input: $input) {
                flow_run_ids
        }
    }
}
variables {
    {"input": {"before": "2020-11-13T09:17:25.861645+00:00", "labels": [], "tenant_id": null}}
service
j
I think you should set it to 
<http://prefect-apollo:4200>
At least, that works for me, also check you are deploying prefect and the agent in the same namespace
m
yes, already deploy in same namespace
Copy code
2020-11-13 09:24:27,121] INFO - agent | Starting KubernetesAgent with labels []
[2020-11-13 09:24:27,121] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-11-13 09:24:27,121] INFO - agent | Agent connecting to the Prefect API at <http://prefect-apollo:4200>
[2020-11-13 09:24:27,132] INFO - agent | Waiting for flow runs...
[2020-11-13 09:24:27,236] ERROR - agent | 400 Client Error: Bad Request for url: <http://prefect-apollo:4200>


This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
    mutation($input: get_runs_in_queue_input!) {
            get_runs_in_queue(input: $input) {
                flow_run_ids
        }
    }
}
variables {
    {"input": {"before": "2020-11-13T09:24:27.134876+00:00", "labels": [], "tenant_id": null}}
j
Ah I see, its connecting. But, honestly i have no idea whats wrong now. Iā€™m new to prefect, maybe someone else has a clue.
m
seems caused i don't have a tenant,
so to create tenant, are you using prefect server installed in your local server? or running in k8s pods?
i'm trying connect to the k8s pods and run the command facing a connection refused
j
Ah,I created the tenant from my local machine too
But again, I donā€™t know the concept of tenant, so not sure whatā€™s best here
m
so, your local machine is running prefect server, right?
j
No I dpeloyed to kubernetes, port forward apollo and the ui, then ran the created the tenant from my local machine
And then deployed the agent, also in kubernetes
m
the port-forward command you running is 
kubectl port-forward service/apollo 4200
?
j
Almost, for me its 
kubectl port-forward svc/prefect-server-apollo 4200
, but that depends on the given name for your deployment. Going of your eerlier screenshot it should be 
svc/prefect-apollo
m
yes, already forwarded
Copy code
kubectl port-forward svc/prefect-apollo 4200
Forwarding from 127.0.0.1:4200 -> 4200
but seems no prefect command installed in my local machine
Copy code
prefect server create-tenant --name default --slug default
-bash: prefect: command not found
j
Ah in order to run such commands, you should have prefect installed
After which you can run prefect commands
m
after run prefect command in local machine, k8s agent can connect to the server and also create new tenant thank you, I very much appreciate your help
j
Great you got it working šŸ™‚
z
Thanks @JoĆ«l Luijmes šŸ™‚
Would expanding the Helm docs to include some port-forward commands be helpful?
Iā€™m considering adding agent support to the Helm chart, not sure if thatā€™d be useful as well.
If anyone would like to trial this or provide feedback as users of the K8s agent: https://github.com/PrefectHQ/server/pull/136
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by