Hi, with my team we are deploying prefect flows that during the data download phase need to be able to be distributed on multiple machines. For parallel execution we had followed this tutorial (https://discourse.prefect.io/t/how-can-i-run-multiple-subflows-or-child-flows-in-parallel/96), but, of course, this solution splits the execution of the subflows always on the same machine (pod). What we wanted instead is that the prefect agent distribute the execution of the subflows on a separate machine since it needs a lot of memory. Kind of what is done with tasks and a dask cluster. I ask, therefore, if there is a way to be able to tell prefect to split each flow onto a new machine. Specifically, we use fargate machines to run the various flows.

Take that subflow and create a deployment with the memory you require. Instead of running the subflows asynchronously on your node, use run_deployment. This works well with AWS ECS.

I agree with Dominic's suggested approach for sure! When you use

run_deployment

for the subflows, they will still show the correct linkages to the parent flow in the flow run graph as well. One way to think about a Prefect deployment is as a combination of a flow and some specification for the infrastructure you want that flow to run on (via the work pool you choose for it)

Thank you all for the response. Yes, we were familiar with deployment and in fact we adopted a similar solution by making N calls to the flows using the rest api "create_flow_run_from_deployment", in this way we achieved parallel execution on multiple machines. However, the various flows were independent and were not linked on the cloud UI, but appeared as N separate and distinct flows. Your proposed solution, on the other hand, solves this problem. We are implementing it at this time. We will let you know whether it is okay or not. Thank you again. @Davide Imperiale @Giovanni Giacco

Ah yes,

create_flow_run_from_deployment

is a lower-level utility that

run_deployment

uses, and then

run_deployment

adds the linking between the parent and subflows.