Hello Let s say I wanted to run multiple dbt commands throug Prefect Community #prefect-server

Hello! Let's say I wanted to run multiple dbt comm...

JC Garcia

11/30/2020, 3:30 PM

Hello! Let's say I wanted to run multiple dbt commands through the dbt task, but will not know the commands until runtime (they will be received as a paramter). What is the recommended way of looping tasks? Would something like this be ok?

Copy code

dbt_task = DbtShellTask(
  ...
)

with Flow("dbt-flow") as flow:

    dbt_command_string = Parameter('dbt_command_string', default = 'dbt run', required = True)

    dbt_commands = dbt_command_string.split(',')


    for command in dbt_commands:
      dbt_task(
          command=dbt_command,
          ...
      )

Zanie

11/30/2020, 4:49 PM

Hi! That won’t work because dbt_commands_string is just a

Parameter

type then, not a string. You’d want to map over the parameter instead.

Zanie

11/30/2020, 4:49 PM

https://docs.prefect.io/core/concepts/mapping.html

JC Garcia

11/30/2020, 4:52 PM

oh that's great! do mapped tasks run sequentially or in parallel?

Zanie

11/30/2020, 6:45 PM

They run in parallel

matta

11/30/2020, 7:12 PM

Is there a way to make them run sequentially? @Zanie

Zanie

11/30/2020, 7:19 PM

You could use a loop e.g. https://docs.prefect.io/core/advanced_tutorials/task-looping.html

Zanie

11/30/2020, 7:45 PM

Additionally, if you just want to leverage the

DbtShellTask

functionality but want to track the running of the commands with a single Prefect task, you can pass the parameter to a task that does split and loop as you have written but call

dbt_task.run(...)

within your task. The benefit of the (more complex)

LOOP

pattern is that each of the subtasks would be tracked by Prefect but if that’s not important to you, calling

.run

directly may be useful.

👍 1

matta

11/30/2020, 7:45 PM

@JC Garcia

👍 1

JC Garcia

12/01/2020, 1:57 AM

thanks @Zanie!

5 Views

Open in Slack

Previous Next