Hey Folks, I'm running into this exception `ValueError: Cannot set

mapped=True

when running from inside a mapped context`` As a dummy example, I'm trying to do something like this:

from prefect.tasks.core.function import FunctionTask
from prefect import Flow, apply_map


def task_block(z):
    R = FunctionTask(lambda x: range(x))(z)
    M = FunctionTask(lambda x: x + 1).map(R)
    return M
    
with Flow('nested_map') as flow:
    L = [1, 2, 3]
    apply_map(task_block, L)

is there a way to do this kind of nested mapping differently?

I don't think I really understand what you are trying for here. Is it: • From a list, generate a list of lists, • Map over each element of the sublist. If that is the case,

flatten

is your friend. https://docs.prefect.io/core/concepts/mapping.html#flat-mapping

Well, here my use case is made dumb for the sake of simplicity. What I m trying to do is to trigger a mapped task within another mapped task. If you want to know my precise use case, imagine a list of videos that I want to chunk in several pieces. So this would look like:

def process_video(video):
    chunks = GetChunksTask()(video) # return a list of tuple (first frame, last frame) for the chunks e.g. [(0, 24), (25, 50)]
    chunked_videos = ChunkVideoTask().map(unmapped(video), chunks)
    return chunked_videos

with Flow('nested_map') as flow:
    videos = ['video_0.mp4', 'video_1.mp4', 'video_2.mp4']
    apply_map(process_video, videos)

Does it make more sense now ?

I see. Unfortunately, nested mapping isn't really supported, at least not yet. I usually handle my nested map needs with some tweaks +

flatten

# Make GetChunksTask return a tuple of (video_name, first_frame, last_frame)
# Note ChunkVideoTask needs to be slightly modified to accept a 3-tuple

from prefect import flatten
 
with Flow('budget_nested_map') as flow:
    videos = ['0', '1']
    chunks = GetChunksTask().map(video) # [[(0, 1, 10), (0,11, 20)], [(0, 1, 20), (1,21, 40)]]
    chunked_videos = ChunkVideoTask().map(flatten(chunks))

This way each of your chunks keep a reference to the actual video, you won't need a nested structure to keep what video has what chunk. Ofc you will need another step to reduce these back into video level, it is what it is 😅

Hi @Romain! @emre is spot on that nested mapping is not supported. The first map already spreads the operation across available resources so a second layer map, doesn’t map sense in terms of parallelization. Users normally use

flatten

to map over a list of list type structure.

Ok I see. I will figure out something using flatten but I believe that having a nested mapping makes the code easier to read and to understand. Anyway thanks to both of you @emre and @Kevin Kho

I will bring up that feedback. You’re the third person who asked this week 🙂

hi @Kevin Kho If I understand correctly, we cannot do neither nested cases right? Because you were speaking about feedbacks, I believe those 2 things (nested maps and nested cases) would defintely make my code cleaner.

You are right that you can’t