How do you go about setting up a volume for jobs when using the kubernetes agent? I see that
--job-template
is an option when using start, but not when using install. I'm able to set up a volume on the agent, but I realized that won't actually make the volumes available to my flows when I run them.
k
Kyle Moon-Wright
03/29/2021, 11:24 PM
Hey @Will Milner,
I think there are a few options available to you.
You can extend the
install
cli command to accommodate a custom
--job-template
to pass to the agent with the volume you’ve determined or use the start command with the
--job-template
flag, potentially pulling a yaml from some cloud storage. Another option would be to attach the custom job template to the KubernetesRun
run_config
of your flow, but only if you’d like everything configured on the flow side.
Overall, the best practice would be to use a ConfigMap that holds the config file and mount it as a volume, but I don’t have much experience going with this option.
w
Will Milner
03/31/2021, 3:24 PM
thanks for the suggestions. I was able to modify the command the agent container uses to accept the --job-template argument, and supplied the template via the volume I had mounted to the agent
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.