Hello, I am using Docker agent to run flows, one of the task inside the flow needs to interact with ...

Prasanth Kothuri

10/29/2021, 5:03 PM

Hello, I am using Docker agent to run flows, one of the task inside the flow needs to interact with k8s cluster which requires KUBECONFIG, currently I am making this available with

--volume

flag, not sure if I am doing it correctly here, what is the best practice ?

Kevin Kho

10/29/2021, 5:05 PM

I think this link will help you with that

Anna Geller

10/29/2021, 5:07 PM

plus, if you just need to trigger a KubernetesJob, you can use one of Kubernetes tasks from the task library, here is an example you could use: https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows_task_library

Kevin Kho

10/29/2021, 5:08 PM

Or maybe this

Prasanth Kothuri

10/29/2021, 5:13 PM

this is how I am starting the agent

Prasanth Kothuri

10/29/2021, 5:14 PM

Copy code

nohup prefect agent docker start --api <http://localhost:4200> --env KUBECONFIG=/root/vk9.config --env S3_ENDPOINT=ZZZ --env S3_ACCESS_KEY=XXX --env S3_SECRET_KEY=YYY --volume /root:/root --show-flow-logs --no-pull 2>&1 > /tmp/prefect.docker.agent.log &

Prasanth Kothuri

10/29/2021, 5:17 PM

flows are able to see the KUBECONFIG and able to create pods, wondering if this is the best way to do this

Kevin Kho

10/29/2021, 5:23 PM

I think it’s fine. I think other ways would require a refactor, and I am not seeing anything wrong here.

👍 1

Prasanth Kothuri

10/29/2021, 5:30 PM

thanks @Kevin Kho one of the step/task in the flow is to run spark_submit on k8s cluster, for this I am using

ShellTask

to do spark-submit, which absolutely works fine ! however would like to know other ways people are doing this

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.