https://prefect.io logo
t

Thomas Opsomer

06/30/2021, 5:49 PM
Hi community 🙂 I’m wondering the best way to handle a long running flow with several tasks having different resource requirement, without wasting too many resources. For instance a flow with 2 tasks that are both quite long: the first task needs many GBs of memory and the second one just a few MBs. Using the k8s agent/executor, prefect is creating a single pod that run the entire flow, so I need to requests the GBs of memory for the whole flow 😕 while it’s needed only for the first task.
k

Kevin Kho

06/30/2021, 5:53 PM
Hi @Thomas Opsomer, typically when you have varying resource requirements for different sections of your flow, the suggestion is to split up the flow in to smaller subflows, triggered from a main flow by using the
StartFlowRun
task.
In this case you can provision different resources for them using the Kubernetes RunConfig. and register them separately, and then call them from the main flow
t

Thomas Opsomer

06/30/2021, 6:22 PM
Ok so that way, I can have a “high level” flow that dispatch each of its subflows and waits until the completion of each subflow. So for the whole flow, I will have 2 pod running: one for the main flow and one for the currently running subflow ?
k

Kevin Kho

06/30/2021, 6:23 PM
Yes but the main flow will be minimal.
t

Thomas Opsomer

07/01/2021, 1:33 PM
Alright, thank you very much !! 🙂
3 Views