Rafał Wojdyła

    Rafał Wojdyła

    1 year ago
    👋 I have a question about Result/Target usage, I would like the target template to be based on the flow parameters - how would I do that? ``````
    j

    josh

    1 year ago
    Hi @Rafał Wojdyła all values that are available in context should be available to the template! One of those being
    parameters
    . So if you have a flow that takes a
    val
    param you should be able to access it through
    {parameters[val]}
    Rafał Wojdyła

    Rafał Wojdyła

    1 year ago
    👋 @josh yes, I have tried that, but run into an issue where default value of parameters are not in the context/parameters. is this by design?
    j

    josh

    1 year ago
    Hmm I just ran a test and I was able to grab the default value from parameters. Target:
    target="{parameters[val]}"
    and the Parameter is
    Parameter("val", default="asdf")
    Rafał Wojdyła

    Rafał Wojdyła

    1 year ago
    @josh which version of Prefect?
    j

    josh

    1 year ago
    I am on the latest version
    Rafał Wojdyła

    Rafał Wojdyła

    1 year ago
    @josh I'm on
    0.11.2
    , could you pls post the code you have used?
    I'm doing something akin to:
    self.location.format(prefect.context)
    where the location has the parameter pattern akin to
    "[parameters[val]}"
    , works when I specify the parameter explicitly, but doesn't for default value (fails with missing key).
    j

    josh

    1 year ago
    Ah it’s possible that old version does not support it (FWIW the current version is
    0.14.7
    ). Here is the code I am using:
    from prefect import Flow, task, Parameter
    from prefect.engine.results import LocalResult
    
    
    @task(target="{parameters[val]}")
    def get_data(val):
        return [val]
    
    
    @task
    def print_data(data):
        print(data)
    
    
    with Flow(
        "using-targets",
        result=LocalResult(),
    ) as flow:
        val = Parameter("val", default="asdf")
        data = get_data(val)
        print_data(data)
    
    flow.run()
    Rafał Wojdyła

    Rafał Wojdyła

    1 year ago
    @josh that was actually the problem (0.11.2 doesn't support defaults it seems). thanks. I have a quick follow up question please, I would like to create dependencies between Flows without registering any of them (which is doc here: https://docs.prefect.io/api/latest/tasks/prefect.html#running-dependent-flows). This is akin to luigi's
    requires
    on another task. Is this something that Prefect supports?
    and just for the context, I'm currently trying to decide whether to use Luigi or Prefect for ETLs (file based) orchestration.
    j

    josh

    1 year ago
    Hmm, without registering them?
    Currently there is a hard requirement that flows (in order to orchestrate with the backend) need to be registered. But that is something that may change in the near future 🙂
    Rafał Wojdyła

    Rafał Wojdyła

    1 year ago
    @josh I see, so to create dependency between flows, right now one must register at least the upstream one (and use
    StartFlowRun
    )?
    j

    josh

    1 year ago
    Yes
    Rafał Wojdyła

    Rafał Wojdyła

    1 year ago
    @josh and if I would like to depend on only one task out of another flow, is there a way to do that?
    and to be clear - that task only makes sense in the context of the original flow (with parameters etc)
    j

    josh

    1 year ago
    Some people achieve that by utilizing targets. For example: Flow A kicks off Flow B. Then B has a task that write some data to a target. Now you can have a task in Flow A that also looks at that same target and can act upon it. Other than that there isn’t yet fully first-class way of doing it
    Rafał Wojdyła

    Rafał Wojdyła

    1 year ago
    @josh I see. thanks all this was very helpful and I think it should be enough for me to make a decision