@Marvin how can I define many subflow runs and then execute them consecutively

Ah, the joy of orchestrating subflows, a task as thrilling as counting grains of sand on a beach. But I digress. Here's how you can do it in Prefect: 1. First, you need to define your subflows. Each subflow is a Flow itself. Here's an example of a simple subflow:

python
from prefect import Flow, task

@task
def subtask1():
    return "Subtask 1 executed"

@task
def subtask2():
    return "Subtask 2 executed"

with Flow("Subflow 1") as subflow1:
    result1 = subtask1()
    result2 = subtask2()

2. You can define as many subflows as you like, just remember to give them unique names. 3. Now, to execute them consecutively, you need to create a parent flow that will run these subflows. You can use the

FlowRunner

task for this:

python
from prefect.engine.flow_runner import FlowRunnerTask

flow_runner = FlowRunnerTask()

with Flow("Parent Flow") as parent_flow:
    subflow1_state = flow_runner.run(flow=subflow1)
    # Add more subflows as needed

4. Finally, you can run the parent flow which will execute the subflows consecutively:

python
parent_flow.run()

Remember, the universe is vast and life is short, so try to make your subflows as efficient as possible. Or don't. I'm a chatbot, not a life coach.

could you just have an iterable of params for your subflow fn, and then do something like for params in subflow_params_list: subflow_fn(params)

Yeah that's what I'm doing, but I'd like to create all the subflow runs upfront without blocking, and then have them execute one at a time. Then if I get a failure on a subflow run and retry, the failed and not started subflows would all run. I think if I do it like this (still testing), a retry would only retry the one failed subflow