Sumit Kumar Rai

    Sumit Kumar Rai

    1 year ago
    Please help me to clear up my confusion on instantiating prefect flows. I have three dbt commands to run:
    dbt deps
    dbt snapshot
    dbt run
    For now I'm running them by reusing the instance of DbtShellTask:
    dbt_task = DbtShellTask(name="Running dbt commands", ...)
    
    dbt_clean_task_invoke   = dbt_task("dbt deps")
    dbt_deps_task_invoke    = dbt_task("dbt snapshot")
    dbt_run_task_invoke     = dbt_task("dbt run")
    With having this setup, my schematic will have three boxes(tasks) with name
    Running dbt commands
    which is difficult to identify which is which. In order to fix that, do I have to instantiate for each command like below.
    dbt_clean_task = DbtShellTask(name="Running dbt deps command", ...)
    dbt_deps_task  = DbtShellTask(name="Running dbt snapshot command", ...)
    dbt_run_task   = DbtShellTask(name="Running dbt run command", ...)
    
    dbt_clean_task_invoke   = dbt_clean_task("dbt deps")
    dbt_deps_task_invoke    = dbt_deps_task("dbt snapshot")
    dbt_run_task_invoke     = dbt_run_task("dbt run")
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Sumit Kumar Rai, I think this is right. If you want it a bit shorter, maybe you can do
    dbt = DbtShellTask
    
    with Flow("a") as flow:
        dbt_clean_task = dbt(name="Running dbt deps command", ...)("dbt deps")
        dbt_deps_task  = dbt(name="Running dbt snapshot command", ...)("dbt snapshot")
        dbt_run_task   = dbt(name="Running dbt run command", ...)("dbt run")
    Matthias Roels

    Matthias Roels

    1 year ago
    You could also do:
    dbt = DbtShellTask(**kwargs)
    with Flow("a") as flow:
        dbt_deps = dbt(command="dbt deps", task_args={"name" = "Running dbt deps command"}
        dbt_snapshot = dbt(command="dbt snapshot", task_args={"name": "Running dbt snapshot command"}, upstream_task=[dbt_deps])
    ...
    Sumit Kumar Rai

    Sumit Kumar Rai

    1 year ago
    Thank you @Kevin Kho and @Matthias Roels for the reply. I followed as @Matthias Roelsā€™s response as it is more readable for me. And the solution worked.