My flow has several

DatabricksRunNow

tasks that run Databricks jobs. The tasks take parameters, which I provide via Prefect’s

Parameter

construct. These parameters that are fed into Databricks tasks always show up in the UI with these extra nodes for

List

and

Dict

. I don’t understand what they’re for, or why I need to see them in the UI, and they really clutter things up. Is this something I can eliminate by restructuring my flow somehow? Or is there some part of Prefect that needs to be changed to eliminate these noisy nodes from the schematic? Or are these

List

and

Dict

nodes actually useful for something?

Hi @Nicholas Chammas - the UI shows those

List

and

Dict

tasks because those are part of your flow; Prefect creates tasks for constant values when they're used in your flow context. That said, I agree they're noisy and we've got some improvements in the works for this display but they're a little ways off. Adjusting them requires a significant rewrite of schematics as they exist now to omit certain tasks as the layout is created based on ids from upstream and downstream dependencies. That said, you can simplify your schematic by moving instances where Prefect would need to generate a task into the tasks themselves, instead of doing so at the flow level. For example:

@task
def task_hello():
  return 'hello'

@task
def task_goodbye():
  return 'goodbye'

@task
def task_b(val):
  for v in val:
    print(v)

with Flow('flow') as flow:
  hello = task_hello()
  goodbye = task_goodbye()

  task_b([hello, goodbye])

could become:

@task
def task_hello():
  return 'hello'

@task
def task_goodbye():
  return 'goodbye'

@task
def task_b(**args):
  for v in args:
    print(v)

with Flow('flow') as flow:
  hello = task_hello()
  goodbye = task_goodbye()

  task_b(hello, goodbye)

@nicholas We would also love to have the decluttering as we use lists and dicts extensively 🙂 Looking forward to seeing it.