https://prefect.io logo
Title
s

Sam Werbalowsky

03/03/2022, 9:40 PM
Continuing to be confused by parameters here…is there any easy way to dynamically use parameters as strings for various task inputs. IE I have a
param=Parameter("myvalue", default="myvalue")
. I want to us it in a variety of tasks say…
execute = execute_file(f"sql/{param}.sql")
upload = upload_file(f"{param}.csv")
Is there a way to do this without constructing a task for each input?
k

Kevin Kho

03/03/2022, 9:42 PM
You need a task because the f-string is evaluated during DAG construction, not runtime so you need to wrap each in a task, but I think you only need to define 1 task?
s

Sam Werbalowsky

03/03/2022, 9:51 PM
well the two strings are different in the input, it’s not just the param
how could that be done with one task?
k

Kevin Kho

03/03/2022, 10:01 PM
Maybe this is unnecessarily complicating things but you can maybe not use f-strings and then make the task take in a
param
and
template
so that you can use string formatting with
template.format(param)
?
You can also wrap your f-string with task and lambda for one line
s

Sam Werbalowsky

03/03/2022, 10:04 PM
Yeah, I was thinking the param, template thing could work. I just wanted to double check that I wasn’t missing anything. It’s a little bit of a bummer parameters are so difficult to work with, as it makes flexible flows (i.e. it mostly does the same thing, just hits different files based on a parameter) a little unwieldy.
Thanks for the help
if you have an example handy with the task/lambda, would love to see it.
k

Kevin Kho

03/03/2022, 10:14 PM
Maybe like this:
from prefect import Flow, task, Parameter
import prefect 

@task
def log(x):
    <http://prefect.context.logger.info|prefect.context.logger.info>(x)
    return 1

with Flow("...") as flow:
    x = Parameter("x", "test")
    log(task(lambda x: f"logging {x}")(x))

flow.run()
Looks ugly in the logs though
And yes but the limitation is parameters are injected in runtime so you need to wrap the string builder in the runtime. Actually we might have a string builder task one sec
ehhh…pretty much the same thing
👍 1