Hey folks we ve been putting together a bunch of custom Task

Question

Hey folks we ve been putting together a bunch of custom Tasks for internal use Lets say one of those is called DoSomething I m wondering what the correct approach is for dealing with Task parameters Providing the task with a Parameter in the flow fails because the task doesn t know what to do with it and calling `run` on the parameter in the flow definition is clearly wrong The Parameter instance s are passed in as a list since the users of the Task need to be able to pass an arbitrary number of unnamed parameters Are there any examples of unpacking the parameters in an efficient way So far I have ```params = param run if isinstance param Parameter else param for param in params ``` I ve also tried running the Parameter through a ` task` function but when I pass the results of that task to DoSomething it complains that FunctionalTask is not compatible

Chris White · Answer

Hi < Jacob Blanco> I m having trouble following the issue you re describing can you produce a self contained code snippet that highlights what you ve tried and the error you get

Jacob Blanco · Answer

```from InternalTasks import DoSomething with Flow my flow as flow DoSomething Parameter my parameter HardcodedParameter ```

Jacob Blanco · Answer

Something like the above < Chris White> I tried wrapping the parameter list creation as a FunctionalTask but I get an error from inside DoSomething that said that FunctionalTask does not have length

Chris White · Answer

Ah OK I see So when you are using the class based approach the first call of the class initializes the task as a task object The second call is the one that is actually deferred and a part of the Flow s task graph ```from prefect import Task Flow Parameter name = Parameter name default= p initialization step with Flow bad flow as flow t = Task name=p initializing Task will fail``` vs ```name = Parameter name default= p initialization step t = Task name= p also an initialization step happens imemdiately with Flow bad flow as flow t upstream tasks= p calling here sets a deferred dependency```

Chris White · Answer

```class MyTask Task def init self kwargs this method always has to run immediately at build time super init kwargs def run self kwargs this method is the one that is deferred and only run at flow run time```

Jacob Blanco · Answer

Just confirming one thing you seem to mix `name` and `p` to mean the Parameter variable there is that right So the correct approach is to initialize the class based Task outside of the flow definition then call the instance of the Task as a function passing in the parameter

Jacob Blanco · Answer

```from InternalTasks import DoSomething do something = DoSomething p = Parameter my parameter with Flow my flow as flow do something p HardcodedParameter ``` Assuming the parameters are taken in from the `run` function in the DoSomething class

Chris White · Answer

Yup exactly

Chris White · Answer

` task` actually initializes a task class under the hood for you passing in any ` kwargs` to the initialization step of the task That s one of the reasons the two APIs can feel quite different

Jacob Blanco · Answer

OK makes sense How does this impact passing the results of an task wrapped step into a class based Task

Chris White · Answer

so you can only pass the results of an ` task` step to an already initialized task ``` task def my task pass within a Flow context Task my task bad t = Task within a Flow context t my task good```

Jacob Blanco · Answer

Gotcha and following that structure as the Task subclass developer I don t need to do anything to unpack the results my task

Chris White · Answer

nope Prefect takes care of making sure the raw return value is passed to the next task

Jacob Blanco · Answer

Awesome sauce thank you

Chris White · Answer

+1 anytime glad I could help

Jacob Blanco · Answer

< Arsenii> This is the solution my great comrade ^

Jacob Blanco · Answer

< Chris White> Related to this I am looking at the Tasks library for inspiration In one of the Postgres task s `run` method you use `self host` instead of host doesn t that mean you will never be able to override it at run time If I get it right the default from attrs decorator replaces the default of the arguments in the function signature for run the attributes of the class set at initialisation right

Chris White · Answer

Yea that s true for that task we could easily tweak that task to accept a `host` at runtime though if you need it

Jacob Blanco · Answer

Oh damn I m silly I didn t look at the function signature you never take `host` in when calling run nevermind

Chris White · Answer

< Marvin> archive Help passing parameters to custom task class

Marvin · Answer