https://prefect.io logo
Title
h

Haotian Li

03/09/2023, 2:28 PM
After updating to 2.8 we found that each sub flow in a parent flow no longer scales to separate Kubernetes pods. Any idea on why this is happening?
r

redsquare

03/09/2023, 2:36 PM
subflow as in you are calling run_deployment?
h

Haotian Li

03/09/2023, 2:44 PM
Subflow as being called in another flow in a deployment. Triggering the deployment through Quick run doesn't scale the subflow
h

Haotian Li

03/09/2023, 2:58 PM
Yes, but the behavior here without using run_deployment and just invoking flow within another flow changed in 2.8, in 2.7 each subflow within a flow occupies a k8s pod by itself.
z

Zanie

03/09/2023, 3:42 PM
Invoking a flow in another flow has never spawned new infrastructure
While I would love to support that, it’s pretty complicated.
h

Haotian Li

03/10/2023, 3:00 PM
So basically every flow will only run on a single kubernetes pod it was created on?
z

Zanie

03/10/2023, 3:12 PM
Unless you do
run_deployment
, yeah.
h

Haotian Li

03/10/2023, 3:24 PM
We would like to make some developmental contributions here to allow for that, is there someone we should talk to in the Prefect team to make sure how we can best work that out?
z

Zanie

03/10/2023, 6:15 PM
How best to work out having subflows automatically run in new pods?
h

Haotian Li

03/11/2023, 3:20 AM
Yes
z

Zanie

03/11/2023, 3:22 AM
I'm the best one to start talking to for something like that.
h

Haotian Li

03/11/2023, 3:22 AM
We meant to do some development in this area and would be happy to potential make a PR. Just want to make sure it's not conflicting with some internal roadmap