Hello, I am using Docker agent to run flows, one o...
# prefect-serverp
Prasanth Kothuri
10/29/2021, 5:03 PMHello, I am using Docker agent to run flows, one of the task inside the flow needs to interact with k8s cluster which requires KUBECONFIG, currently I am making this available with
--volumek
Kevin Kho
10/29/2021, 5:05 PM
I think this link will help you with that
a
Anna Geller
10/29/2021, 5:07 PM
plus, if you just need to trigger a KubernetesJob, you can use one of Kubernetes tasks from the task library, here is an example you could use: https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows_task_library
k
Kevin Kho
10/29/2021, 5:08 PM
Or maybe this
p
Prasanth Kothuri
10/29/2021, 5:13 PMthis is how I am starting the agent
Prasanth Kothuri
10/29/2021, 5:14 PM Copy code
nohup prefect agent docker start --api <http://localhost:4200> --env KUBECONFIG=/root/vk9.config --env S3_ENDPOINT=ZZZ --env S3_ACCESS_KEY=XXX --env S3_SECRET_KEY=YYY --volume /root:/root --show-flow-logs --no-pull 2>&1 > /tmp/prefect.docker.agent.log &Prasanth Kothuri
10/29/2021, 5:17 PMflows are able to see the KUBECONFIG and able to create pods, wondering if this is the best way to do this
k
Kevin Kho
10/29/2021, 5:23 PM
I think it’s fine. I think other ways would require a refactor, and I am not seeing anything wrong here.
👍 1
p
Prasanth Kothuri
10/29/2021, 5:30 PMthanks @Kevin Kho one of the step/task in the flow is to run spark_submit on k8s cluster, for this I am using
ShellTask2 Views