https://prefect.io logo
f

Fred Birchall

12/13/2022, 4:04 PM
Hey Everyone! Does somebody know if we can
.submit
a task and use one of the parameters as the task name? I know we can currently do
my_task.submit(name="My Task")
and this will show up as
my_task-a1b2c3-0
in the UI, but what I would like to do is:
Copy code
from prefect import flow, task

@flow(name="My Flow")
def my_flow():
    my_task("Hello")
    my_task(name="World")

@task(name_from_param="name")
def my_task(name):
    pass
Then in the Prefect UI I will see:
Copy code
Running Flow: My Flow
Running Task: Hello-a1b2c3-0
Running Task: World-a1b2c3-1
A bit of background for context, we run all of our tasks either using AWS Lambda or DBT. For Lambda, I’ve built a central function (which is decorated as a task) with a signature
run_lambda(lambda_name, payload={})
, and most users are using it like
run_lambda.submit("MyLambdaFunction")
but then in the logs all the task names are
run_lambda-[ascii]-0
which is annoying when things go pop, as you have to click into the task run where I have added
<http://logger.info|logger.info>("Lambda name: %s", lambda_name)
for traceability. For all of my personal Flows I have the following convention
run_lambda.with_options(name="Lambda MyLambdaFunction").submit("MyLambdaFunction")
, which gives me the desired Prefect Task names in the UI, but getting everyone else to follow suit is another matter. I’ve attempted many solutions to dynamically name a task mainly focused on custom decorators, but they have all failed in one way or another… So I wanted to see if the Prefect community has any ideas or solutions! Thanks
k

Khuyen Tran

12/13/2022, 4:29 PM
This might be something you are looking for
r

Ryan Peden

12/13/2022, 4:39 PM
It sounds like
with_options
does the job, but asking the team to run this one task differently than other tasks is the issue.
☝️ 1
f

Fred Birchall

12/14/2022, 10:13 AM
Hey @Khuyen Tran - Thanks for the link, I’m already using
.with_options
extensively myself, but I’m looking for an option to dynamically name a task using function parameters. I’ve taken a look at the source code of the
task
decorator: tasks.py#L158-L164, and I see that the name of the Task is defined before the function is called and thus before the function parameters are defined. Further exploring the source code, it looks like
enter_task_run_engine
is the central function that all Task invocations pass through: engine.py#L898. Feel like that around Line 897, there could be something like:
Copy code
if task.name_from_param is not None:
    task.name = parameters.get(task.name_from_param, task.name)
k

Khuyen Tran

12/14/2022, 5:09 PM
@Fred Birchall I still try to understand what you mean by dynamically name a task using function parameters. Based on your provided example, wouldn’t doing this work?
Copy code
@flow(name="My Flow")
def my_flow():
    task1.with_options(name="Hello")()
    task1.with_options(name="World")()
f

Fred Birchall

12/15/2022, 9:27 AM
@Khuyen Tran Yes that example would work, but for my use case we have a library of functions that is available in all Prefect runtimes so that each developer does not have to re-invent the wheel with every flow. One such function looks like:
Copy code
@task
def run_lambda(lambda_name, event={}):
   [...]
Most of our developers are not too experienced in Python so this library of tools also features as a way to abstract some of the complexities, like
.with_options
. So my ultimate goal is to allow our developers to run
run_lambda("MyFunction")
and the task in the UI to be called
MyFunction-a1b2c3-0
without our developers having to worry about the
.with_options
method.
k

Khuyen Tran

12/15/2022, 5:11 PM
You can’t currently create a custom name for a task without a
with_options
. You might consider open an issue on GitHub for this.
l

Leonardus Chen

01/27/2023, 2:59 AM
Reviving an old thread here, in Prefect 1.0, we can use this to achieve the above:
Copy code
@task(task_run_name="{param1}")
def say_hello(param1: str) -> None:
    print(param1)
Is there no such feature in Prefect 2.0 yet?
3 Views