Hey all, @Adithya Ramanathan and I are running into some issues with Prefect’s mapping functionality and would appreciate some more information on the mechanics of mapping in Prefect! My understanding is that currently, having two consecutive mapped components allows you to run the second component on the same level of mapping as the first component, like in one of the examples in the docs:

from prefect import Flow, task

@task
def add_ten(x):
    return x + 10

with Flow('iterated map') as flow:
    mapped_result = add_ten.map([1, 2, 3])
    mapped_result_2 = add_ten.map(mapped_result)

This allows you to map the first add_ten call to the three elements in the list, and then to map the second add_ten call to the three outputs from the first call. Instead of returning a single int from the add_ten function, we want it to return an list of ints and continue to fan-out (to 6 elements in the example below):

@task
def add_ten(x):
    return [x + 10, x]

with Flow('iterated map') as flow:
    mapped_result = add_ten.map([1, 2, 3])
    mapped_result_2 = add_ten.map(mapped_result)

Is this something that is supported by Prefect, or can you only have 1 fan-out per map/reduce?

You're looking for "flat mapping", which lets you flatten an iterable of iterables into a single iterable. See https://docs.prefect.io/core/concepts/mapping.html#flat-mapping for more info.

Great, thanks Jim! This is helpful. I’ll have to play around with that, it seems very promising for our use-case!