https://prefect.io logo
r

Rafał Wojdyła

02/10/2021, 5:16 PM
👋 I have a question about Result/Target usage, I would like the target template to be based on the flow parameters - how would I do that? ``````
j

josh

02/10/2021, 5:19 PM
Hi @Rafał Wojdyła all values that are available in context should be available to the template! One of those being
parameters
. So if you have a flow that takes a
val
param you should be able to access it through
{parameters[val]}
r

Rafał Wojdyła

02/10/2021, 5:21 PM
👋 @josh yes, I have tried that, but run into an issue where default value of parameters are not in the context/parameters. is this by design?
j

josh

02/10/2021, 5:28 PM
Hmm I just ran a test and I was able to grab the default value from parameters. Target:
target="{parameters[val]}"
and the Parameter is
Parameter("val", default="asdf")
r

Rafał Wojdyła

02/10/2021, 5:28 PM
@josh which version of Prefect?
j

josh

02/10/2021, 5:28 PM
I am on the latest version
r

Rafał Wojdyła

02/10/2021, 5:29 PM
@josh I'm on
0.11.2
, could you pls post the code you have used?
I'm doing something akin to:
self.location.format(prefect.context)
where the location has the parameter pattern akin to
"[parameters[val]}"
, works when I specify the parameter explicitly, but doesn't for default value (fails with missing key).
j

josh

02/10/2021, 5:33 PM
Ah it’s possible that old version does not support it (FWIW the current version is
0.14.7
). Here is the code I am using:
Copy code
from prefect import Flow, task, Parameter
from prefect.engine.results import LocalResult


@task(target="{parameters[val]}")
def get_data(val):
    return [val]


@task
def print_data(data):
    print(data)


with Flow(
    "using-targets",
    result=LocalResult(),
) as flow:
    val = Parameter("val", default="asdf")
    data = get_data(val)
    print_data(data)

flow.run()
r

Rafał Wojdyła

02/10/2021, 5:44 PM
@josh that was actually the problem (0.11.2 doesn't support defaults it seems). thanks. I have a quick follow up question please, I would like to create dependencies between Flows without registering any of them (which is doc here: https://docs.prefect.io/api/latest/tasks/prefect.html#running-dependent-flows). This is akin to luigi's
requires
on another task. Is this something that Prefect supports?
and just for the context, I'm currently trying to decide whether to use Luigi or Prefect for ETLs (file based) orchestration.
j

josh

02/10/2021, 5:46 PM
Hmm, without registering them?
Currently there is a hard requirement that flows (in order to orchestrate with the backend) need to be registered. But that is something that may change in the near future 🙂
r

Rafał Wojdyła

02/10/2021, 5:49 PM
@josh I see, so to create dependency between flows, right now one must register at least the upstream one (and use
StartFlowRun
)?
j

josh

02/10/2021, 5:50 PM
Yes
r

Rafał Wojdyła

02/10/2021, 5:50 PM
@josh and if I would like to depend on only one task out of another flow, is there a way to do that?
and to be clear - that task only makes sense in the context of the original flow (with parameters etc)
j

josh

02/10/2021, 5:51 PM
Some people achieve that by utilizing targets. For example: Flow A kicks off Flow B. Then B has a task that write some data to a target. Now you can have a task in Flow A that also looks at that same target and can act upon it. Other than that there isn’t yet fully first-class way of doing it
r

Rafał Wojdyła

02/10/2021, 5:58 PM
@josh I see. thanks all this was very helpful and I think it should be enough for me to make a decision