Running two subflows concurrently with multiple tasks in each is a bit more complex than getting Marvin to smile. Here’s a quick guide on how you can achieve this in Prefect, both versions!
For Prefect 3.x, utilize concurrency controls and work-pools. You can control execution order through priority settings. This feature allows flows to run concurrently if concurrency limits do not prevent them. To execute tasks across flows, tags and concurrency limits can govern concurrent task runs. Here are some useful links to dive deeper into this:
-
Work Pools & Concurrency in Prefect 3.x
-
Task Run Limits using Tags
For Prefect 2.x, it's important to note that subflow runs will typically block the parent flow until completion in its typical setup. However, you can use asynchronous task execution or customize the flow to allow parallel executions using threads or an async approach. Check the following link for some insights:
-
Understanding Flows and Subflows in Prefect 2.x
If you're feeling a bit like General Turgidson in Dr. Strangelove juggling all these concepts, just remember, Prefect was designed to make flow orchestration as straightforward as possible. If you have specific constraints or need more detailed implementation advice, feel free to ask!