https://prefect.io logo
Join Slack
Powered by
I have a question about dynamically generating tas...
# ask-community
b
I have a question about dynamically generating tasks. 🧵
✅ 1
Here is some sample code:
Copy code
from prefect.tasks.jupyter.jupyter import ExecuteNotebook

def generate_notebook_tasks(flow):
    for nb_file in os.listdir():
        run_nb_task = ExecuteNotebook(
            path=nb_file,
            parameters={}
        )
        flow.add_task(run_nb_task)


def create_flow() -> Flow:
    with Flow() as flow:
        generate_notebook_tasks(flow=flow)
        return flow


FLOW = create_flow()
Is it reasonable to generate tasks like above at flow generation time, or is it better to have a task that returns a collection of tasks.
n
instead of using the imperative API to create a flow like above, I would just map 
ExecuteNotebook
over your 
nb_files
like
Copy code
from prefect.tasks.jupyter.jupyter import ExecuteNotebook

execute_notebook_task = ExecuteNotebook() # add default params here

with Flow('my nb execute flow') as flow:

   # could also do nb_files = glob.glob("*.ipynb")
   nb_files = [nb_file for nb_file in os.listdir()] 

   results = execute_notebook_task.map(path=nb_files)
prefect will generate an instance of 
execute_notebook_task
for each element in your iterable 
nb_files
b
Thanks @Nate - The problem for mapping is that we don’t end up with a schematic that shows each task, right?
Ideally, I would end up with something like:
n
visually in the schematic, the mapped task would be collapsed into one item, but it shows how many mapped instances have been created. you can click on that item and it'll show mapped task runs.
b
Thanks - I appreciate the feedback!
👍 1
Open in Slack
PreviousNext
Powered by