Nicholas Chammas
06/30/2021, 6:15 PMDatabricksRunNow
tasks that run Databricks jobs. The tasks take parameters, which I provide via Prefect’s Parameter
construct.
These parameters that are fed into Databricks tasks always show up in the UI with these extra nodes for List
and Dict
. I don’t understand what they’re for, or why I need to see them in the UI, and they really clutter things up.
Is this something I can eliminate by restructuring my flow somehow? Or is there some part of Prefect that needs to be changed to eliminate these noisy nodes from the schematic? Or are these List
and Dict
nodes actually useful for something?nicholas
06/30/2021, 6:36 PMList
and Dict
tasks because those are part of your flow; Prefect creates tasks for constant values when they're used in your flow context.
That said, I agree they're noisy and we've got some improvements in the works for this display but they're a little ways off. Adjusting them requires a significant rewrite of schematics as they exist now to omit certain tasks as the layout is created based on ids from upstream and downstream dependencies.
That said, you can simplify your schematic by moving instances where Prefect would need to generate a task into the tasks themselves, instead of doing so at the flow level. For example:
@task
def task_hello():
return 'hello'
@task
def task_goodbye():
return 'goodbye'
@task
def task_b(val):
for v in val:
print(v)
with Flow('flow') as flow:
hello = task_hello()
goodbye = task_goodbye()
task_b([hello, goodbye])
could become:
@task
def task_hello():
return 'hello'
@task
def task_goodbye():
return 'goodbye'
@task
def task_b(**args):
for v in args:
print(v)
with Flow('flow') as flow:
hello = task_hello()
goodbye = task_goodbye()
task_b(hello, goodbye)
Thomas Hoeck
08/05/2021, 1:12 PM