hello everyone, I'm new to prefect, just started hacking today. I'm trying to use it to create dependency for my databricks jobs. But I can't seem to figure out how to create dependency between jobs. and I keep getting this error. (is imperative API the only way?). I'm running it with an agent in a VM if that's relevant

You can also use the upstream_tasks kwarg while calling a task, within a flow context. In your case: p2 = RunNow(job_id=... , upstream_tasks=[p1]) I don't know why your flow is failing though.

Thank you for the reply I think it probably have something to do with how

DatabrickRunNow

implemented, especially it seems that

RunNow(job_id=<another task's output>)

is problematic. if I pass a static variable, it works. to save some headache. I rolled out my own function with databricks job API, works now. will investigate later see if I missed something.

Hey @dex, it’s not that a task output is problematic. It’s more of the Dict access is problematic (and List will be too). Think of it that the

name_job_map

happens with deferred execution while the Dict access is done immediately when that object does not exist quite yet (it is still a Task class). You can do

task(lambda x: x['pipe1'])(name_job_map)

instead of

name_job_map['pipe_1']

, and I think that should solve it since we move that to deferred execution also

@Kevin Kho thank you for the answer : )