https://prefect.io
Join Slack
Hello! Let's say I wanted to run multiple dbt commands through the <dbt task>, but will not know the...
# prefect-server
j
Hello! Let's say I wanted to run multiple dbt commands through the dbt task, but will not know the commands until runtime (they will be received as a paramter). What is the recommended way of looping tasks? Would something like this be ok?
Copy code
dbt_task = DbtShellTask(
  ...
)

with Flow("dbt-flow") as flow:

    dbt_command_string = Parameter('dbt_command_string', default = 'dbt run', required = True)

    dbt_commands = dbt_command_string.split(',')


    for command in dbt_commands:
      dbt_task(
          command=dbt_command,
          ...
      )
z
Hi! That won’t work because dbt_commands_string is just a 
Parameter
type then, not a string. You’d want to map over the parameter instead.
j
oh that's great! do mapped tasks run sequentially or in parallel?
z
They run in parallel
m
Is there a way to make them run sequentially? @Zanie
z
Additionally, if you just want to leverage the 
DbtShellTask
functionality but want to track the running of the commands with a single Prefect task, you can pass the parameter to a task that does split and loop as you have written but call 
dbt_task.run(...)
within your task. The benefit of the (more complex) 
LOOP
pattern is that each of the subtasks would be tracked by Prefect but if that’s not important to you, calling 
.run
directly may be useful.
👍 1
m
@JC Garcia
👍 1
j
thanks @Zanie!
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by