https://prefect.io logo
j

Joël Luijmes

02/18/2021, 2:59 PM
Hey, I’m looking for something similar as asked here (batch mapped tasks or nested mapping). Has anyone got something like this working? The example provided by Michael does something differently than I want. What I’m after is: 1. Retrieve some dynamic list (i.e. query from database) 2. Batch the result set in 20ish items to process in parallel 3. For each item in the batch, do a branch of chained tasks 4. Wait for batch to complete, repeat for next batch until exhausted Basically I want to write something as
Copy code
tasks = dynamic_list_of_tasks()
windowed_tasks = fixed_window(tasks, window_size=5)

def process_item(item):
    x = task_1()
    task_2(x)

def process_window(tasks):
    apply_map(process_item, tasks)

apply_map(process_window, windowed_tasks)
@Josh Greenhalgh you ever got something like this working?
j

Josh Greenhalgh

02/18/2021, 3:00 PM
Unfortunately not...
😞 1
I think its very much an important feature though!
j

Joël Luijmes

02/18/2021, 3:02 PM
Thanks for replying! Yes agreed, would be great if it was possible to have nested mappings (or batch process of a mapping)
z

Zanie

02/19/2021, 5:41 PM
Sorry that we haven't given an answer here in this thread 🙂 I think responding to the Github issue is the best bet as this is something that will take significant consideration for us to include full support for.
j

Joël Luijmes

02/19/2021, 7:59 PM
Thats okay 🙂. I kinda concluded that too, it seems like a nice feature. However, since it isn’t supported yet, it seems like might be tricky to implement. Everyone, rally behind this ticket: https://github.com/PrefectHQ/prefect/issues/2459 😛