Hello! We have a flow that is highly resource intensive. We already specify custom resource requests but that alone is not enough. When the agent schedules it; we would like it to make use of a priority class to ensure that it has access to all the resources it needs. How can this be done? I see in the docs that there is a way to provide a custom job template:
But I’m not sure how to write that template / how this template is used (are there any docs on this?). Thanks!
k
Kevin Kho
05/18/2022, 2:53 PM
So the template is loaded in and then used and merged with the other settings you provide. If you don’t provide a job template, there is actually a default one used here. I guess this is a starting template, the requirement is you need a container named
flow
Kevin Kho
05/18/2022, 2:53 PM
The agent needs access to that s3 address during runtime
j
Jake
05/18/2022, 2:55 PM
this is exactly what I was looking for thanks!
Jake
05/18/2022, 2:56 PM
If I provide the job_template; the do the other settings get merged in (labels, flow metadata, resource requests etc.) if they are not part of the template?
k
Kevin Kho
05/18/2022, 2:59 PM
Yes they do. Priority given to stuff in KubernetesRun, they will override the template specifications
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.