Hey all :slightly_smiling_face: We got the following error when trying to map over a list of dictio...

Robin

10/19/2020, 4:15 PM

Hey all 🙂 We got the following error when trying to map over a list of dictionaries. First just to clarify: mapping over a list of dictionaries should be fine, right? Second, what could be the reason for the message?

✔️ 1

Jim Crist-Harif

10/19/2020, 4:25 PM

Hi Robin, mapping over a list of dictionaries should be fine. The error you're getting here is happening in your task here (the actual mapping appears to be working correctly) - looks like you're getting a list where you expected a dict - perhaps you're passing in a list of lists instead?

Robin

10/19/2020, 4:52 PM

Good point! Thanks for the quick feedback 🙂

Robin

10/19/2020, 5:17 PM

In that case, this should work, right?

Copy code

reduce_list = task(lambda list_of_lists: reduce(list.extend, list_of_lists, []))

Robin

10/19/2020, 7:17 PM

Copy code

@task
def reduce_list(list_of_lists):
    out = []
    for l in list_of_lists:
        out.extend(l)
        
    return out

did the trick 🙂

Jim Crist-Harif

10/19/2020, 8:38 PM

That'd work, you could also make use of the new(ish)

prefect.flatten

, which will do that for you (but potentially in a more efficient way).

Copy code

your_task.map(flatten(list_of_lists))

See https://docs.prefect.io/core/concepts/mapping.html#flat-mapping for more info.

Robin

10/19/2020, 9:06 PM

Thanks for the tip! that looks neat 🙂

Robin

10/19/2020, 9:06 PM

@Severin Ryberg [sevberg]

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.