for some reason when using a function in ` task` arguments t Prefect Community #ask-community

for some reason , when using a function in `@task`...

David Migl

10/10/2023, 12:29 AM

for some reason , when using a function in

@task

arguments, there are different behaviors when using a function that has arguments vs when using a function that does not: •

dynamic_naming_function_with_no_arguments

--> invoke function in task arguments, i.e

@task(task_run_name = dynamic_naming_function_with_no_arguments)

--> using

prefect.runtime.task_run.get_parameters()

will get the parameters from a task as expected. •

dynamic_naming_function_with_arguments(str1: str, str2: str)

--> invoke function in task arguments, i.e

@task(task_run_name = dynamic_naming_function with_arguments(arg1,arg2)

--> using

prefect.runtime.task_run.get_parameters()

will return

{}

task_run

values associated with

prefect.runtime

library are not populated when using a function that takes in arguments and I am wondering about this behavior and if there was any explanation for it. I appreciate your time!

David Migl

10/10/2023, 12:39 AM

for context prefect version 2.13.1

Ryan Peden

10/10/2023, 6:37 AM

The first thing that comes to mind is when you do:

Copy code

@task(task_run_name = dynamic_naming_function_with_no_arguments)

task_run_name

is a reference to the function, which Prefect can then use to call the function when it creates a task run for this task. But when you call

Copy code

@task(task_run_name = dynamic_naming_function with_arguments(arg1,arg2))

Python runs the function immediately when it loads the script, and sets

task_run_name

to whatever the function returned at that time. So it's not really a Prefect thing, it's just how Python works.

dynamic_naming_function_with_no_arguments

doesn't call the function immediately, but if you had done

@task(task_run_name = dynamic_naming_function_with_no_arguments())

, it would have failed in the same way as the

@task(task_run_name = dynamic_naming_function with_arguments(arg1,arg2)

call. If you want it to work with those args you could wrap the call in a lambda:

Copy code

@task(task_run_name = lambda: dynamic_naming_function with_arguments(arg1,arg2))

This will read arg1 and arg2 every time the function runs, and should work as long as arg1 and arg2 are in scope and have their values set before you run this task. You could also do:

Copy code

from functools import partial

@task(task_run_name=partial(dynamic_naming_function, arg1, arg2))

or:

Copy code

def dynamic_naming_function with_arguments(arg1, arg2):
    def wrapper():
        # put all your original naming code inside the wrapper function

    return wrapper

# and then

@task(task_run_name=dynamic_naming_function_with_arguments(arg1, arg2))

The last two example are a little different from the first in that they'll always use whatever the value of arg1 and arg2 were when the script first ran. If the values of arg1 and arg2 change later, your naming function won't see those changes. If the values of arg1 and arg2 never change, it doesn't matter much which one you use.

❤️ 1

Ryan Peden

10/10/2023, 7:05 AM

Finally, it looks like any callable will work for task_run_name, not just a function. So if you ever want fancy, encapsulated, verbose task run namer, you could also do something like this:

Copy code

class GreatTaskRunNamer:
    def __init__(self, arg1, arg2):
        self.arg1 = arg1
        self.arg2 = arg2

    def __call__(self):
        task_run_args = runtime.task_run.get_parameters()
        task_run_name = f"{self.arg1}-{self.arg2}-{task_run_args['task_arg_1']}-{task_run_args['task_arg_2']}"

        return task_run_name

@task(task_run_name=GreatTaskRunNamer(arg1, arg2))
def my_amazingly_named_task(task_arg_1, task_arg_2):
   print("hello from the best task ever")

Here, the initial call to

GreatTaskRunNamer(arg1, arg2)

calls `GreatTaskRunNamer`'s constructor (

__init__

) and passes in arg1 and arg2, which then get saved for later. And then, every time the task runs, it invokes `GreatTaskRunNamer`'s

__call__

function to generate the task run name. In case this seems pointless, I promise this example actually serves a purpose! For anyone reading this who has written lots of classes but not many functions-that-return-functions or partials: under the hood, this isn't much different that what the wrapped function and

partial

call are doing. After all, objects are a poor man's closures (and vice versa).

❤️ 1

David Migl

10/13/2023, 8:01 PM

Hey @Ryan Peden, I really appreciate the response.

4 Views

Open in Slack

Previous Next