https://prefect.io logo
Title
p

Prasanth Kothuri

10/29/2021, 5:03 PM
Hello, I am using Docker agent to run flows, one of the task inside the flow needs to interact with k8s cluster which requires KUBECONFIG, currently I am making this available with
--volume
flag, not sure if I am doing it correctly here, what is the best practice ?
k

Kevin Kho

10/29/2021, 5:05 PM
I think this link will help you with that
a

Anna Geller

10/29/2021, 5:07 PM
plus, if you just need to trigger a KubernetesJob, you can use one of Kubernetes tasks from the task library, here is an example you could use: https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows_task_library
k

Kevin Kho

10/29/2021, 5:08 PM
Or maybe this
p

Prasanth Kothuri

10/29/2021, 5:13 PM
this is how I am starting the agent
nohup prefect agent docker start --api <http://localhost:4200> --env KUBECONFIG=/root/vk9.config --env S3_ENDPOINT=ZZZ --env S3_ACCESS_KEY=XXX --env S3_SECRET_KEY=YYY --volume /root:/root --show-flow-logs --no-pull 2>&1 > /tmp/prefect.docker.agent.log &
flows are able to see the KUBECONFIG and able to create pods, wondering if this is the best way to do this
k

Kevin Kho

10/29/2021, 5:23 PM
I think it’s fine. I think other ways would require a refactor, and I am not seeing anything wrong here.
👍 1
p

Prasanth Kothuri

10/29/2021, 5:30 PM
thanks @Kevin Kho one of the step/task in the flow is to run spark_submit on k8s cluster, for this I am using
ShellTask
to do spark-submit, which absolutely works fine ! however would like to know other ways people are doing this