https://prefect.io logo
Title
j

JC Garcia

11/30/2020, 3:30 PM
Hello! Let's say I wanted to run multiple dbt commands through the dbt task, but will not know the commands until runtime (they will be received as a paramter). What is the recommended way of looping tasks? Would something like this be ok?
dbt_task = DbtShellTask(
  ...
)

with Flow("dbt-flow") as flow:

    dbt_command_string = Parameter('dbt_command_string', default = 'dbt run', required = True)

    dbt_commands = dbt_command_string.split(',')


    for command in dbt_commands:
      dbt_task(
          command=dbt_command,
          ...
      )
z

Zanie

11/30/2020, 4:49 PM
Hi! That won’t work because dbt_commands_string is just a
Parameter
type then, not a string. You’d want to map over the parameter instead.
j

JC Garcia

11/30/2020, 4:52 PM
oh that's great! do mapped tasks run sequentially or in parallel?
z

Zanie

11/30/2020, 6:45 PM
They run in parallel
m

matta

11/30/2020, 7:12 PM
Is there a way to make them run sequentially? @Zanie
z

Zanie

11/30/2020, 7:19 PM
Additionally, if you just want to leverage the
DbtShellTask
functionality but want to track the running of the commands with a single Prefect task, you can pass the parameter to a task that does split and loop as you have written but call
dbt_task.run(...)
within your task. The benefit of the (more complex)
LOOP
pattern is that each of the subtasks would be tracked by Prefect but if that’s not important to you, calling
.run
directly may be useful.
👍 1
m

matta

11/30/2020, 7:45 PM
@JC Garcia
👍 1
j

JC Garcia

12/01/2020, 1:57 AM
thanks @Zanie!