Thread
#prefect-server
    p

    Prasanth Kothuri

    10 months ago
    Hello, I am using Docker agent to run flows, one of the task inside the flow needs to interact with k8s cluster which requires KUBECONFIG, currently I am making this available with
    --volume
    flag, not sure if I am doing it correctly here, what is the best practice ?
    Kevin Kho

    Kevin Kho

    10 months ago
    I think this link will help you with that
    Anna Geller

    Anna Geller

    10 months ago
    plus, if you just need to trigger a KubernetesJob, you can use one of Kubernetes tasks from the task library, here is an example you could use: https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows_task_library
    Kevin Kho

    Kevin Kho

    10 months ago
    Or maybe this
    p

    Prasanth Kothuri

    10 months ago
    this is how I am starting the agent
    nohup prefect agent docker start --api <http://localhost:4200> --env KUBECONFIG=/root/vk9.config --env S3_ENDPOINT=ZZZ --env S3_ACCESS_KEY=XXX --env S3_SECRET_KEY=YYY --volume /root:/root --show-flow-logs --no-pull 2>&1 > /tmp/prefect.docker.agent.log &
    flows are able to see the KUBECONFIG and able to create pods, wondering if this is the best way to do this
    Kevin Kho

    Kevin Kho

    10 months ago
    I think it’s fine. I think other ways would require a refactor, and I am not seeing anything wrong here.
    p

    Prasanth Kothuri

    10 months ago
    thanks @Kevin Kho one of the step/task in the flow is to run spark_submit on k8s cluster, for this I am using
    ShellTask
    to do spark-submit, which absolutely works fine ! however would like to know other ways people are doing this