n

    Nicholas Chammas

    1 year ago
    If I pass a
    Parameter
    directly to a
    DatabricksRunNow
    task, Prefect detects the dependency and passes the information from the parameter to the task correctly. e.g.
    path = Parameter("path")
    
    DatabricksRunNow(...)(
        databricks_conn_secret=SECRET,
        notebook_params: {
            "path": path,
        }.
    )
    However, if I plug the parameter into a formatted string, for example, the information is no longer passed from the parameter to the task correctly.:
    path = Parameter("path")
    
    DatabricksRunNow(...)(
        databricks_conn_secret=SECRET,
        notebook_params: {
            "path": f"{path}",
        }.
    )
    In this case, the parameter class instance is plugged into the string vs. the actual parameter value that we want. So the notebook is given a
    repr()
    of a
    Parameter
    class instance — which is unusable, of course — instead of the string value of the parameter that we actually want. Why is that, and is there a way around this?
    My use case: I am trying to build a string out of flow parameter values and pass that string as an input notebook parameter. I could change the notebook to accept the raw parameter values and build the string inside the notebook, so I can do that if necessary. It just seemed more natural in my case to build the string as part of the flow pass the already-built string to the notebook.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Nicholas Chammas, I think you need an intermediate task to do this because that f-string is evaluated during build time and you need something evaluated at runtime.
    Maybe you can do something like
    "path": task(lambda x: f"{x}")(path)
    n

    Nicholas Chammas

    1 year ago
    Hmm, OK. Just so I understand: How come that’s not necessary in the first example? Is this where Prefect is doing the extra work by generating those intermediate
    List
    and
    Dict
    tasks?
    Kevin Kho

    Kevin Kho

    1 year ago
    The Parameter is a task also so it’s also evaluated at runtime. Stuff like list access or dictionary access
    (x[0], x["test"])
    are also evaluated immediately like the
    f-string
    so the execution of these is not deferred so you can’t do these in the Flow block if
    x
    is the output of a task inside the Flow.