I have a

Parameter

being fed into a

DatabricksRunNow

Task

as a

notebook_param

. I can see the edge between the parameter and the task in the UI fine, but for whatever reason the Databricks notebook is not receiving the parameter value when I run the flow. What should I try to troubleshoot the reason why the notebook task is not receiving parameter values?

Hi @Nicholas Chammas, could you show me the code of the

DatabricksRunNow

task and how you pass it in? Just remove any sensitive stuff.

The flow code is roughly this:

def databricks_job(job_id, **kwargs):
    if "name" in kwargs:
        name_prefix = f"{kwargs['name']} - "
    else:
        name_prefix = ""
    kwargs.setdefault("task_run_name", f"{name_prefix}databricks_job[{job_id}]")
    kwargs.update(job_id=job_id)
    return DatabricksRunNow(**kwargs)


GenerateCounts = databricks_job(
    name="Generate Counts",
    job_id=GENERATE_COUNTS_JOB_ID,
)


with le_flow(name="...", state_handlers=[HANDLER]) as flow:
    version = Parameter("version", required=True)

    generate_counts = GenerateCounts(
        databricks_conn_secret=DATABRICKS_CONN,
        notebook_params={
            "attemptId": version,
        },
    )

It looks fine to me and the source code looks fine. Does it work if you don’t use the param? I assume

notebook_params

does not get passed?

Hmm, it looks like if I replace

version

inside

"attemptId": version,

with a plain string, the notebook still doesn’t get any parameters. I guess the problem is not the parameter then! That’s an obvious test I should have tried. 😅 Guess I need to understand what’s going wrong with

notebook_params

to begin with.

Does your job go through though? Meaning the connection secret got used?

Yes, the job is triggered but always with no/empty notebook parameter values.

Stupid question, but just making sure you’re re-registering right? This seems weird notebook_params is not respected. What version are you on?

I’m on Prefect 0.14.22. The reregistration is working. If I provide the

notebook_params

as static kwargs (i.e. alongside things like

name

and

job_id

) it works, but when I provide them as runtime/dynamic

kwargs

that’s when they are being ignored. I’ll try

Task.run()

and report back.

Very weird because the code seems to just grab the init arguments and use them at runtime if there are done provided

OK, I think I found the issue. Our ECS Prefect agent was running Prefect 0.14.12. Updating that to 0.14.22 fixed the problem. I got suspicious when running the flow locally (per your suggestion) worked, but running it on our deployed ECS agent didn’t.

Ohh wow. Good to know!