https://prefect.io logo
Title
t

Tim Enders

08/16/2022, 4:40 PM
It looks like I can no longer have keyword function params defaulted to
None
defined on a mapped task? I now get this error (
TypeError: object of type 'NoneType' has no len()
) when trying to map across the subsequent function signature.
@task
def get_items_list(
    client, entity, page,
    response_attr=None, path=None, params=None
):
1
1
If I remove the keyword args I don't get the error, however I kind of need those there for functionality
a

Anna Geller

08/16/2022, 11:02 PM
can you show how you use it with mapping? props to you if you could build an example we could use to reproduce I believe the issue may just be that the argument you iterate over must be a list and all others must be wrapped in unmapped
t

Tim Enders

08/17/2022, 1:26 PM
Using it with a mapping would be as follows. This would use the default
None
values defined in the function signature:
items = get_items_list.map(
        unmapped(client), unmapped("subscriptions"), pages_list
    )
a

Anna Geller

08/18/2022, 12:19 AM
not sure but you may need to switch the order of arguments:
items = get_items_list.map(
        pages_list, unmapped(client), unmapped("subscriptions")
    )
t

Tim Enders

08/18/2022, 1:30 PM
No, there are 3 positional arguments (required) and 3 keyword arguments (optional). However, it seems I need to make the keyword arguments default to an empty list
[]
for the mapping to work. Which makes the code smelly to me, and doesn't seem like very good Python.
a

Alex Wilcoxson

08/18/2022, 7:43 PM
The issue I saw when debugging is that since the task is mapped, Prefect is checking the len on each argument (as all the params should have equal len when mapping. Since the kwarg is omitted and its default value has no length, it throws So to work around it I've specified all optional kwargs as unmapped(...) It would be nice if Prefect could infer that because the optional is not provided, the default should be implicitly unmapped.
💯 1
t

Tim Enders

08/18/2022, 8:18 PM
Prefect 1.0 did. I think this is a bug?
a

Anna Geller

08/19/2022, 1:59 AM
thanks to you both for bringing this up and describing the issue - I'll investigate more and report back
I looked more and I believe this behavior is expected and even really useful -- there seems to be a tradeoff between allowing default unmapped parameters or allowing to map over multiple values simultaneously, and we went for the second option because the first can be easily solved by passing
unmapped(None)
in place of those default unmapped arguments as shown here:
"""
with: mapped_future = add_one.map([1, 2, 3])
prefect.exceptions.MappingLengthMismatch: Received parameters with different lengths.
Parameters for map must all be the same length. Got lengths: {'x': 3, 'statis_value': 6}
"""
from prefect import flow, task, get_run_logger, unmapped


@task
def add_one(x: int, statis_value: str = None) -> int:
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello, %s!", statis_value)
    <http://logger.info|logger.info>("Current value of x is %s", x)
    return x + 1


@flow
def mapping_unmapped():
    mapped_future = add_one.map([1, 2, 3], unmapped(None))
    # mapped_future = add_one.map([1, 2, 3], unmapped("Anna"))
    # add_one.map(mapped_future, unmapped("Second mapped loop"))


if __name__ == "__main__":
    mapping_unmapped()
t

Tim Enders

08/19/2022, 2:30 PM
Thank you!
a

Anna Geller

08/19/2022, 2:47 PM
@Tim Enders + @Alex Wilcoxson I wasn't entirely wrong but talking more to engineers looks like we will do exactly what you suggested i.e. we'll treat default values as
unmapped
to keep things consistent for the caller of
map
- there is an open issue for that here https://github.com/PrefectHQ/prefect/issues/6383 Again, thanks so much for reporting this
a

Alex Wilcoxson

08/19/2022, 2:54 PM
cool, thanks!
🙌 1