https://prefect.io logo
Title
b

Bradley Hurley

09/06/2022, 9:07 PM
I have a question about dynamically generating tasks. 🧵
1
Here is some sample code:
from prefect.tasks.jupyter.jupyter import ExecuteNotebook

def generate_notebook_tasks(flow):
    for nb_file in os.listdir():
        run_nb_task = ExecuteNotebook(
            path=nb_file,
            parameters={}
        )
        flow.add_task(run_nb_task)


def create_flow() -> Flow:
    with Flow() as flow:
        generate_notebook_tasks(flow=flow)
        return flow


FLOW = create_flow()
Is it reasonable to generate tasks like above at flow generation time, or is it better to have a task that returns a collection of tasks.
n

Nate

09/06/2022, 9:45 PM
instead of using the imperative API to create a flow like above, I would just map
ExecuteNotebook
over your
nb_files
like
from prefect.tasks.jupyter.jupyter import ExecuteNotebook

execute_notebook_task = ExecuteNotebook() # add default params here

with Flow('my nb execute flow') as flow:

   # could also do nb_files = glob.glob("*.ipynb")
   nb_files = [nb_file for nb_file in os.listdir()] 

   results = execute_notebook_task.map(path=nb_files)
prefect will generate an instance of
execute_notebook_task
for each element in your iterable
nb_files
b

Bradley Hurley

09/06/2022, 10:54 PM
Thanks @Nate - The problem for mapping is that we don’t end up with a schematic that shows each task, right?
Ideally, I would end up with something like:
n

Nate

09/06/2022, 11:03 PM
visually in the schematic, the mapped task would be collapsed into one item, but it shows how many mapped instances have been created. you can click on that item and it'll show mapped task runs.
b

Bradley Hurley

09/06/2022, 11:25 PM
Thanks - I appreciate the feedback!
👍 1