When using a KubernetesExecutor, is it possible to have dynamic resource allocation?
We can currently use the
job_spec
file to specify the K8 resources. But we have a use case where, in most situations the volume of work expected from our Flow fits within the specified resources, occasionally an input Parameter to the Flow will be such that it requires more resources. We're getting by this today by setting the resources to the maximum value, but (in theory) that restricts the number of concurrent Flows we can execute in the same environment.
Curious if there's an ability to adjust the memory resources required in the
job_spec
based on the value of an input Parameter to the FlowRun?
j
Jeremiah
05/29/2020, 1:08 AM
That’s a great question. I don’t know the answer offhand but I know there’s been talk around lifecycle hooks for environments, perhaps this ties in
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.