https://prefect.io logo
Channels
Powered by
a

Adrian Fernandez

01/17/2023, 10:34 PM
Hi 👋 , what's the recommended approach to map some arguments and unmapp others for a task that takes kwargs as argument? I'm getting a 
MappingLengthMismatch
error while doing this (more details on the thread) and I have the impression that kwargs can't be 
unmapped
, neither explicitly nor implicitly. My current work around is to wrap the desired task (A) with another task that doesn't take kwargs and call task a as a function 
task_a.fn()
.
plus one 1
My environment:
Copy code
Python 3.8.16
prefect 2.7.8
calling 
add_xy()
without kwargs
Copy code
from prefect import task, flow, unmapped


@task
def add_xy(x: int, y: int, **kwargs):
    print("Kwargs:", kwargs)
    return x + y


@flow
def my_flow():
    results = add_xy.map(x=[1, 2], y=unmapped(3))


print(my_flow())
# MappingLengthMismatch: Received iterable parameters with different lengths.
# Parameters for map must all be the same length. Got lengths: {'x': 2, 'kwargs': 0}
Calling 
add_xy()
with unmapped kwargs
Copy code
@flow
def my_flow_2():
    results = add_xy.map(x=[1, 2], y=unmapped(3), kwargs=unmapped(None))


print(my_flow_2())
# MappingLengthMismatch: Received iterable parameters with different lengths.
# Parameters for map must all be the same length. Got lengths: {'x': 2, 'kwargs': 1}
Work around: add a wrapper task
Copy code
@task
def add_wrapper(x: int, y: int):
    return add_xy.fn(x=x, y=y)


@flow
def my_flow_3():
    results = add_wrapper.map(x=[1, 2], y=unmapped(3))
    print(results)


print(my_flow_3())
# [Completed(message=None, type=COMPLETED, result=4), Completed(message=None, type=COMPLETED, result=5)]
What's the proper way to do this?
z

Zanie

01/17/2023, 10:42 PM
Hi! Thanks for pointing this out. This is a bug in our implementation.
🫢 1
It’s a little tricky to address though.. hm.
a

Adrian Fernandez

01/17/2023, 10:47 PM
Ok, then in the meantime I'll keep the workaround in place
Thanks for the quick response
b

Ben Ayers-Glassey

01/17/2023, 10:58 PM
@Zanie could 
inspect.signature
be used for this?
Copy code
import inspect

def some_func(x, /, posonly, *args, kwonly, **kwargs): ...

sig = inspect.signature(some_func)

assert sig.parameters['kwargs'].kind == inspect.Parameter.VAR_KEYWORD

for param in sig.parameters.values(): print(f"{param.name}: {param.kind}")
^ That outputs:
Copy code
x: POSITIONAL_ONLY
posonly: POSITIONAL_OR_KEYWORD
args: VAR_POSITIONAL
kwonly: KEYWORD_ONLY
kwargs: VAR_KEYWORD
https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object ...or maybe this is tricky for some other reasons. But it seems to me like parameters of kind VAR_KEYWORD should always be treated as unmapped?.. Edit: or in any case, the dict itself should be treated as unmapped. 🤔
z

Zanie

01/17/2023, 11:00 PM
Well, you’d probably expect us to map over a key in kwargs if it is iterable
b

Ben Ayers-Glassey

01/17/2023, 11:03 PM
Yeah. But basically it seems like right now the 
kwargs
dict itself is being checked for iterability, because I believe this error occurs even if you don't pass any kwargs. So ideally, you'd want the values of the dict to be checked for iterability (or for whether they were marked with 
unmapped
). But certainly an empty dict should be allowed! Otherwise, AFAICT you can't easily write a task which wraps another one (unless you explicitly write out all its parameters in the wrapping function's definition, and explicitly pass them all along).
z

Zanie

01/17/2023, 11:18 PM
There’s still a bug here that I can’t dig into right this moment, but this is the approach I’m thinking https://github.com/PrefectHQ/prefect/pull/8188
🙌 1
b

Ben Ayers-Glassey

01/17/2023, 11:19 PM
Wow, I'm going to second @Adrian Fernandez’s comment: quick response! I'll have a look at this, and thank you very much 🙂
z

Zanie

01/17/2023, 11:25 PM
Ah I found the bug it was dumb — it should be working! Let me know if you can break it or if you want to add any test cases 🙂
🙌 2
4 Views
Powered by