Hi community 🙂 I’m wondering the best way to handle a long running flow with several tasks having different resource requirement, without wasting too many resources. For instance a flow with 2 tasks that are both quite long: the first task needs many GBs of memory and the second one just a few MBs. Using the k8s agent/executor, prefect is creating a single pod that run the entire flow, so I need to requests the GBs of memory for the whole flow 😕 while it’s needed only for the first task.

Hi @Thomas Opsomer, typically when you have varying resource requirements for different sections of your flow, the suggestion is to split up the flow in to smaller subflows, triggered from a main flow by using the

StartFlowRun

task.

Ok so that way, I can have a “high level” flow that dispatch each of its subflows and waits until the completion of each subflow. So for the whole flow, I will have 2 pod running: one for the main flow and one for the currently running subflow ?

Yes but the main flow will be minimal.

Alright, thank you very much !! 🙂