Hey, how to do nested mapping?

Hi, @Issam Assafi! I not sure, but it seems like nested mapping is not supported. Check this question: https://prefect-community.slack.com/archives/CL09KU1K7/p1628064350289600

I see, i have this piece of code :

page_path_pdf_pairs = get_pages_from_pdf(pdf_path)
        texts_of_pages = get_text_from_page.map(page_path_pdf_pairs)
        ocr_pages = join_ocr_pages(texts_of_pages)

but in the code above it's applied only to 1 pdf, and i have multiple PDFs ... any idea how i can handle this while gaining parallelisationadvantage?

You can use

flatten

to, well, flatten your nested lists into a single list, which will then be mappable.

from prefect import flatten

with Flow('x') as f:
    page_path_pdf_pairs = get_pages_from_pdf.map(pdf_paths)
    texts_of_pages = get_text_from_page.map(flatten(page_path_pdf_pairs))
    ocr_pages = join_ocr_pages(texts_of_pages)

Be careful, by using flatten you are losing the nested structure, so you might lose the information of which page belongs to which pdf. Tag your pages with their source pdf names wherever possible, so you can group the pages back.

Nested mapping is not a thing because the first level map already uses the available resources, so the inner map wouldn’t really do anything. Emre’s suggestion might be the way to go.

Well.. that kinda depends on how many items are in the first level map

Ah I see what you mean. If you share a code snippet, I could try and help you get it to one map?

not really code, but:

It certainly is. I have been looking at the mapping code lately myself. Will keep this in mind. Are you on a DaskExecutor?

Probably we'll use Dask on ECS.. not sure yet