https://prefect.io logo
Title
m

Mathijs Miermans

03/24/2022, 4:41 PM
Is
flatten
expected to work on a
dict
?
from prefect import Flow, task, context, flatten

@task
def A():
    return {i: list(range(i)) for i in range(3)}

@task
def B(y):
    logger = context.get("logger")
    <http://logger.info|logger.info>(y)


with Flow('flat map') as f:
    a = A()  # {0: [], 1: [0], 2: [0,1]}
    b = B.map(flatten(a))

if __name__ == "__main__":
    f.run()
I'm getting an unexpected error:
ERROR - prefect.flat map | Unexpected error occured in FlowRunner: KeyError(3)
If not, my workaround would be:
@task
def flatten_dict(d):
    return [(k, v) for k, v in d.items()]
s

Steve R

03/24/2022, 4:43 PM
that doesn't seem like it would do what flatten is supposed to do?
:upvote: 1
m

Mathijs Miermans

03/24/2022, 4:44 PM
I might be missing something. I'm trying to map over all key-value pairs in the dict.
s

Steve R

03/24/2022, 4:46 PM
then i don't see that you need to "flatten" the dict, just return your list(dict.items())
k

Kevin Kho

03/24/2022, 4:46 PM
Steve is right. Flatten is to bump nested lists into a one-level list
m

Mathijs Miermans

03/24/2022, 4:51 PM
Sorry, my own example didn't do what I wanted. 🤦‍♂️ This is what I'm looking for:
@task
def flatten_dict(d):
    return [(k, v) for k, v_list in d.items() for v in v_list]
It converts
{0: [], 1: [0], 2: [0,1]}
to
[(1, 0), (2, 0), (2, 1)]
k

Kevin Kho

03/24/2022, 4:52 PM
Ahh, Are you good now?
m

Mathijs Miermans

03/24/2022, 4:52 PM
Yeah, just wanted to confirm that
flatten
doesn't cover this.
k

Kevin Kho

03/24/2022, 4:53 PM
Yep that’s right
👍 1