Mathijs Miermans

    Mathijs Miermans

    6 months ago
    Is
    flatten
    expected to work on a
    dict
    ?
    from prefect import Flow, task, context, flatten
    
    @task
    def A():
        return {i: list(range(i)) for i in range(3)}
    
    @task
    def B(y):
        logger = context.get("logger")
        <http://logger.info|logger.info>(y)
    
    
    with Flow('flat map') as f:
        a = A()  # {0: [], 1: [0], 2: [0,1]}
        b = B.map(flatten(a))
    
    if __name__ == "__main__":
        f.run()
    I'm getting an unexpected error:
    ERROR - prefect.flat map | Unexpected error occured in FlowRunner: KeyError(3)
    If not, my workaround would be:
    @task
    def flatten_dict(d):
        return [(k, v) for k, v in d.items()]
    Steve R

    Steve R

    6 months ago
    that doesn't seem like it would do what flatten is supposed to do?
    Mathijs Miermans

    Mathijs Miermans

    6 months ago
    I might be missing something. I'm trying to map over all key-value pairs in the dict.
    Steve R

    Steve R

    6 months ago
    then i don't see that you need to "flatten" the dict, just return your list(dict.items())
    Kevin Kho

    Kevin Kho

    6 months ago
    Steve is right. Flatten is to bump nested lists into a one-level list
    Mathijs Miermans

    Mathijs Miermans

    6 months ago
    Sorry, my own example didn't do what I wanted. 🤦‍♂️ This is what I'm looking for:
    @task
    def flatten_dict(d):
        return [(k, v) for k, v_list in d.items() for v in v_list]
    It converts
    {0: [], 1: [0], 2: [0,1]}
    to
    [(1, 0), (2, 0), (2, 1)]
    Kevin Kho

    Kevin Kho

    6 months ago
    Ahh, Are you good now?
    Mathijs Miermans

    Mathijs Miermans

    6 months ago
    Yeah, just wanted to confirm that
    flatten
    doesn't cover this.
    Kevin Kho

    Kevin Kho

    6 months ago
    Yep that’s right