Hi I need all of the tasks in the flow to have some shared c

Question

Hi I need all of the tasks in the flow to have some shared configuration so I decided to follow into Prefect example and create a custom decorator But I have problem with correctly forwarding the arguments to the task EDIT details code and stackstrackes moved to the thread Any suggestions how to make it work

Jenny · Answer

Hi < Krzysztof Nawara> it s hard to know but looks like there s too many arguments somewhere Can you share the code you using when you take out args And could you also please move the long error messages into this thread so we can then more easily and also so they don t block up the main thread Thanks

Krzysztof Nawara · Answer

If I understand it correct since there is no args in task siguanture even 1 positional argument is too much And when I build my flow I m using positional arguments I ll try to build minimal example and post it here

Krzysztof Nawara · Answer

Details Code I started with ``` curry def wrapped task fn Callable task init kwargs Any gt prefect tasks core function FunctionTask def wrapped fn args kwargs return fn args kwargs wrapped fn name = fn name t = prefect tasks core function FunctionTask fn=wrapped fn task init kwargs return t``` But this fails Prefect signature validation ```Traceback most recent call last File lt frozen importlib bootstrap gt line 991 in find and load File lt frozen importlib bootstrap gt line 975 in find and load unlocked File lt frozen importlib bootstrap gt line 671 in load unlocked File lt frozen importlib bootstrap external gt line 783 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File Users knawara fun analytics mllib v1 prefect utils crossvalidation py line 10 in lt module gt def generate cv splits data k=7 rep count=1 File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 site packages toolz functoolz py line 303 in call return self partial args kwargs File Users knawara fun analytics mllib v1 prefect utils flow py line 23 in wrapped task t = prefect tasks core function FunctionTask fn=wrapped fn task init kwargs File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 site packages prefect tasks core function py line 60 in init prefect core task validate run signature fn type ignore File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 site packages prefect core task py line 52 in validate run signature raise ValueError ValueError Tasks with variable positional arguments args are not supported because all Prefect arguments are stored as keywords As a workaround consider modifying the run method to accept kwargs and feeding the values to args ``` So I tried without args but then it fails on bind ```Traceback most recent call last File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 contextlib py line 131 in exit self gen throw type value traceback File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 site packages prefect core flow py line 342 in flow context yield self File Users knawara fun analytics mllib v1 prefect utils flow py line 81 in flow builder task configurator mode=mode File Users knawara fun analytics titanic pipeline analysis flow py line 213 in configure tasks prepared data = wrap task prepare data kwargs raw data File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 site packages prefect core task py line 470 in call new bind File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 site packages prefect core task py line 511 in bind callargs = dict signature bind args kwargs arguments type Dict File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 inspect py line 3025 in bind return self bind args kwargs File Users knawara Applications miniconda3 envs titanic pipeline lib python3 8 inspect py line 2951 in bind raise TypeError TypeError too many positional arguments```

Jenny · Answer

Thanks < Krzysztof Nawara> if you do a reproducible example you might want to post in our it s easier for us to track issues there

Krzysztof Nawara · Answer

Okay I ve got the MRE First the case of 2 independent tasks works perfectly ```import prefect from toolz import curry from typing import Callable Any curry def wrapped task fn Callable task init kwargs Any gt prefect tasks core function FunctionTask def wrapped fn kwargs print wrapped return fn kwargs wrapped fn name = fn name t = prefect tasks core function FunctionTask fn=wrapped fn task init kwargs return t from prefect import Flow wrapped task def t1 print executing t1 return 10 wrapped task def t2 print executing t2 return 20 with Flow tf as tf t1 t2 s = tf run ```

Krzysztof Nawara · Answer

Now when I introduce dependencies I get the abovementioned errors ```import prefect from toolz import curry from typing import Callable Any curry def wrapped task fn Callable task init kwargs Any gt prefect tasks core function FunctionTask def wrapped fn args kwargs print wrapped return fn args kwargs wrapped fn name = fn name t = prefect tasks core function FunctionTask fn=wrapped fn task init kwargs return t from prefect import Flow wrapped task def t1 print executing t1 return 10 wrapped task def t2 t1 print executing t2 return t1 2 with Flow tf as tf t2 t1 s = tf run ```

Chris White · Answer

Hi Krzysztof and thanks for the code example as the error states Prefect does not support arbitrary positional arguments usually denoted ` args` Only a fixed set of positional arguments or keyword arguments are supported Your `wrapped fn` signature contains ` args` and this is where the error is originating from

Krzysztof Nawara · Answer

That s correct But if I remove args and leave only kwargs I get 2nd error too many positional arguments What signature a generic wrapper should have

Chris White · Answer

you re going to have to do some metaprogramming on the signature of your `wrapped fn` unfortunately right now your wrapped function only takes keyword arguments but your task accepts both a single positional argument as well as a keyword argument There are ways of updating the signature of the `wrapped fn` to match that of the provided `fn` but you ll have to learn about ` signature ` and `inspect` Assuming you can document this for yourself and collaborators a workaround is to always use keyword arguments when binding tasks ```with Flow tf as tf t2 t1=t1 ```

Krzysztof Nawara · Answer

That s what I was afraid of I ll have to look at it Thanks slightly smiling face

Krzysztof Nawara · Answer

Apparently Python is more than happy to do the heavy lifting for me <https stackoverflow com a 147878 12640922> ``` functools wraps fn def wrapped fn args kwargs print shared storage return fn args kwargs ```

Chris White · Answer

Ahhh good call