Hello everyone! I hope you're doing well. I have ...
# ask-community
o
Hello everyone! I hope you're doing well. I have a question about Prefect version 3.0.10 and would really appreciate your help. I'm working on creating a group of workers (pods) to share the load of a complex flow. The flow consists of sub-flows, and each sub-flow has multiple tasks, like this: - Parent Flow --| Sub Flow 1 ----| Task 1 (runs 10 times, ideally on different machines in parallel) ----| Task 2 --| Sub Flow 2 ----| Task 1 ----| Task 3 The flow itself is quite long, but it runs at a low frequency. Right now, I'm using the
serve
function with the following code:
Copy code
def serve_multiple_flows(list_of_flows: List[Flow], concurrent_limit: int = 10):
    list_of_deployments = [flow.to_deployment(name=flow.name) for flow in list_of_flows]
    serve(*list_of_deployments, limit=concurrent_limit)
However, I’ve noticed that all the sub-flows and tasks run on the same worker that got the Parent Flow, and the other workers aren't being utilized. Is there a built-in solution to distribute the workload across workers? I'd like to avoid creating another subprocess to serve tasks using
prefect.task_worker.serve
and calling them with
delay()
from the sub-flows. Thank you so much in advance!
prefect rocket 1