https://prefect.io logo
m

Max Eggers

08/31/2023, 1:28 PM
@Marvin what are best practices, if any, for using run_deployment to kick off subflows from within a flow
m

Marvin

08/31/2023, 1:29 PM
Here are some best practices for using
run_deployment
to kick off subflows from within a flow: 1.
run_deployment
is used to run a pre-existing deployment as a subflow of the calling flow. This is effectively the same as calling the flow object itself in a parent flow with something like
asyncio.gather
. The big difference is that when you use
run_deployment
, the subflows will run according to the infrastructure associated with that deployment, whereas when you call the
subflow()
object itself, it will necessarily run on the calling (parent) flow’s infrastructure as a process. [^1^] 2. You can run multiple flows from deployments concurrently with no issues. Here's an example code snippet:
Copy code
python
from prefect.deployments import run_deployment

for _ in range(1, 100):
    run_deployment(name="flow/deployment", timeout=0)
This is a workaround, and it's recommended to try this for now. [^2^] Please note that these are excerpts from discussions and may not represent the official documentation. Always refer to the official Prefect documentation for the most accurate and up-to-date information. [^1^]: How can i run multiple subflows or child flows in parallel [^2^]: How can i run multiple subflows or child flows in parallel