Thread
#prefect-community
    Will Milner

    Will Milner

    1 year ago
    is creating tasks in a loop a good idea? say I have something like
    for x in range(3):
       task = some_task(x)
    
    final_task = another_task(upstream_tasks=task)
    I see 3 tasks get created in the loop, but for the final task it only has 1 upstream task, instead of all the tasks created in the loop
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey Will, any reason you need a loop as opposed to mapping? https://docs.prefect.io/core/concepts/mapping.html
    Thinking the example under “Prefect approach” may help you.
    Will Milner

    Will Milner

    1 year ago
    mapping could work, I'm trying to pass the result into a ShellTask, though I suppose I could convert the task into python and invoke the shell from there
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Will Milner, maybe something like this would help this use case
    from prefect import Flow
    from prefect.tasks.shell import ShellTask
    
    sql_list = ["1.sql", "2.sql", "3.sql"]
    shell_task = ShellTask(
        helper_script="",
        shell="bash",
        log_stderr=True,
        return_all=True,
        stream_output=True,
    )
    with Flow(name="Example") as flow:
        tasks = [
            shell_task(
                command="/opt/prefect_env/bin/python /path/to//SnowSQL.py {}".format(
                    statement
                )
            )
            for statement in sql_list
        ]
        for i in range(1, len(tasks)):
            tasks[i].set_upstream(tasks[i - 1])
    flow.run()
    This loops through tasks while setting the upstream dependency
    Will Milner

    Will Milner

    1 year ago
    this is perfect thank you