Ken Nguyen

    Ken Nguyen

    5 months ago
    If I have 2 parameters, one as a list to be mapped, and one as a value that should be unmapped, how can I concat the mapped parameters onto the unmapped one?
    For example I tried something like this:
    with Flow("data-quality-tracking-parallel-dbt-run-flow", run_config=RUN_CONFIG, storage=STORAGE, executor=LocalDaskExecutor()) as flow:
        flags_to_add = Parameter('flags_to_add', default=["--flag1", "--flag2"], required=False)
        dbt_command = Parameter('dbt_command', default="dbt run")
    
        run_dbt.map(
            command=str(unmapped(dbt_command)) + flags_to_add,
        )
    But got the following error:
    TypeError: can only concatenate str (not "list") to str
    Kevin Kho

    Kevin Kho

    5 months ago
    You have to do it in a task, this is a build time vs run time thing. str on unmapped is just a str on a task class so it’s not what you want
    Anna Geller

    Anna Geller

    5 months ago
    I think you need something like this - as Kevin mentioned, a separate task that will generated a list of commands for mapping based on your input parameter values passed as data dependencies:
    from prefect import task, Parameter, Flow
    from typing import List
    from prefect.tasks.dbt.dbt import DbtShellTask
    
    run_dbt = DbtShellTask(...)
    
    
    @task
    def get_full_dbt_commands(
        flags_to_add: List[str], dbt_command: str = "dbt run"
    ) -> List[str]:
        return [f"{dbt_command} {flag}" for flag in flags_to_add]
    
    
    with Flow("data-quality-tracking-parallel-dbt-run-flow") as flow:
        dbt_flags = Parameter("flags_to_add", default=["--flag1", "--flag2"])
        dbt_command = Parameter("dbt_command", default="dbt run")
        dbt_commands = get_full_dbt_commands(dbt_flags, dbt_command)
    
        run_dbt.map(command=dbt_commands)
    I think you don't need
    unmapped
    because you want to iterate over this command in your mapping, correct?
    Ken Nguyen

    Ken Nguyen

    5 months ago
    Thank you both!