https://prefect.io
Join Slack
I have a question about dynamically generating tasks. :thread:
# ask-community
b
I have a question about dynamically generating tasks. 🧵
✅ 1
Here is some sample code:
Copy code
from prefect.tasks.jupyter.jupyter import ExecuteNotebook

def generate_notebook_tasks(flow):
    for nb_file in os.listdir():
        run_nb_task = ExecuteNotebook(
            path=nb_file,
            parameters={}
        )
        flow.add_task(run_nb_task)


def create_flow() -> Flow:
    with Flow() as flow:
        generate_notebook_tasks(flow=flow)
        return flow


FLOW = create_flow()
Is it reasonable to generate tasks like above at flow generation time, or is it better to have a task that returns a collection of tasks.
n
instead of using the imperative API to create a flow like above, I would just map 
ExecuteNotebook
over your 
nb_files
like
Copy code
from prefect.tasks.jupyter.jupyter import ExecuteNotebook

execute_notebook_task = ExecuteNotebook() # add default params here

with Flow('my nb execute flow') as flow:

   # could also do nb_files = glob.glob("*.ipynb")
   nb_files = [nb_file for nb_file in os.listdir()] 

   results = execute_notebook_task.map(path=nb_files)
prefect will generate an instance of 
execute_notebook_task
for each element in your iterable 
nb_files
b
Thanks @Nate - The problem for mapping is that we don’t end up with a schematic that shows each task, right?
Ideally, I would end up with something like:
n
visually in the schematic, the mapped task would be collapsed into one item, but it shows how many mapped instances have been created. you can click on that item and it'll show mapped task runs.
b
Thanks - I appreciate the feedback!
👍 1
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by