https://prefect.io
Join Slack
is creating tasks in a loop a good idea? say I have something like ```for x in range(3): task = s...
# ask-community
w
is creating tasks in a loop a good idea? say I have something like
Copy code
for x in range(3):
   task = some_task(x)

final_task = another_task(upstream_tasks=task)
I see 3 tasks get created in the loop, but for the final task it only has 1 upstream task, instead of all the tasks created in the loop
k
Hey Will, any reason you need a loop as opposed to mapping? https://docs.prefect.io/core/concepts/mapping.html
Thinking the example under “Prefect approach” may help you.
w
mapping could work, I'm trying to pass the result into a ShellTask, though I suppose I could convert the task into python and invoke the shell from there
k
Hey @Will Milner, maybe something like this would help this use case
Copy code
from prefect import Flow
from prefect.tasks.shell import ShellTask

sql_list = ["1.sql", "2.sql", "3.sql"]
shell_task = ShellTask(
    helper_script="",
    shell="bash",
    log_stderr=True,
    return_all=True,
    stream_output=True,
)
with Flow(name="Example") as flow:
    tasks = [
        shell_task(
            command="/opt/prefect_env/bin/python /path/to//SnowSQL.py {}".format(
                statement
            )
        )
        for statement in sql_list
    ]
    for i in range(1, len(tasks)):
        tasks[i].set_upstream(tasks[i - 1])
flow.run()
This loops through tasks while setting the upstream dependency
👍 1
w
this is perfect thank you
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by