Fred Birchall
12/13/2022, 4:04 PM.submit
a task and use one of the parameters as the task name? I know we can currently do my_task.submit(name="My Task")
and this will show up as my_task-a1b2c3-0
in the UI, but what I would like to do is:
from prefect import flow, task
@flow(name="My Flow")
def my_flow():
my_task("Hello")
my_task(name="World")
@task(name_from_param="name")
def my_task(name):
pass
Then in the Prefect UI I will see:
Running Flow: My Flow
Running Task: Hello-a1b2c3-0
Running Task: World-a1b2c3-1
A bit of background for context, we run all of our tasks either using AWS Lambda or DBT. For Lambda, I’ve built a central function (which is decorated as a task) with a signature run_lambda(lambda_name, payload={})
, and most users are using it like run_lambda.submit("MyLambdaFunction")
but then in the logs all the task names are run_lambda-[ascii]-0
which is annoying when things go pop, as you have to click into the task run where I have added <http://logger.info|logger.info>("Lambda name: %s", lambda_name)
for traceability. For all of my personal Flows I have the following convention run_lambda.with_options(name="Lambda MyLambdaFunction").submit("MyLambdaFunction")
, which gives me the desired Prefect Task names in the UI, but getting everyone else to follow suit is another matter. I’ve attempted many solutions to dynamically name a task mainly focused on custom decorators, but they have all failed in one way or another… So I wanted to see if the Prefect community has any ideas or solutions! ThanksKhuyen Tran
12/13/2022, 4:29 PMRyan Peden
12/13/2022, 4:39 PMwith_options
does the job, but asking the team to run this one task differently than other tasks is the issue.Fred Birchall
12/14/2022, 10:13 AM.with_options
extensively myself, but I’m looking for an option to dynamically name a task using function parameters. I’ve taken a look at the source code of the task
decorator: tasks.py#L158-L164, and I see that the name of the Task is defined before the function is called and thus before the function parameters are defined. Further exploring the source code, it looks like enter_task_run_engine
is the central function that all Task invocations pass through: engine.py#L898. Feel like that around Line 897, there could be something like:
if task.name_from_param is not None:
task.name = parameters.get(task.name_from_param, task.name)
Khuyen Tran
12/14/2022, 5:09 PM@flow(name="My Flow")
def my_flow():
task1.with_options(name="Hello")()
task1.with_options(name="World")()
Fred Birchall
12/15/2022, 9:27 AM@task
def run_lambda(lambda_name, event={}):
[...]
Most of our developers are not too experienced in Python so this library of tools also features as a way to abstract some of the complexities, like .with_options
. So my ultimate goal is to allow our developers to run run_lambda("MyFunction")
and the task in the UI to be called MyFunction-a1b2c3-0
without our developers having to worry about the .with_options
method.Khuyen Tran
12/15/2022, 5:11 PMwith_options
. You might consider open an issue on GitHub for this.Leonardus Chen
01/27/2023, 2:59 AM@task(task_run_name="{param1}")
def say_hello(param1: str) -> None:
print(param1)
Is there no such feature in Prefect 2.0 yet?