Hi everyone,
I have a big flow with hundreds of smaller tasks that run in parallel. Currently, a flow run is processed in only one k8s pod. Is it possible to process those tasks in multiple pods at the same time?
✅ 1
d
Deceivious
05/31/2023, 9:36 AM
Might be a usecase of sub flows.
j
Jozo
05/31/2023, 9:39 AM
I've tried subflows but they run in the same pod as the parent flow. Is it correct behavior or should Prefect create new k8s jobs/pods for subflows?
d
Deceivious
05/31/2023, 9:40 AM
If you directly call subflows it runs in the same env as the parent.
But You can use have the subflow as a seperate deployment and call run_deployments method to have those subflows running as a seperate flow in new pods.