Will Milner
03/24/2021, 5:45 PMfor x in range(3):
task = some_task(x)
final_task = another_task(upstream_tasks=task)
I see 3 tasks get created in the loop, but for the final task it only has 1 upstream task, instead of all the tasks created in the loopKevin Kho
Kevin Kho
Will Milner
03/24/2021, 7:35 PMKevin Kho
Kevin Kho
from prefect import Flow
from prefect.tasks.shell import ShellTask
sql_list = ["1.sql", "2.sql", "3.sql"]
shell_task = ShellTask(
helper_script="",
shell="bash",
log_stderr=True,
return_all=True,
stream_output=True,
)
with Flow(name="Example") as flow:
tasks = [
shell_task(
command="/opt/prefect_env/bin/python /path/to//SnowSQL.py {}".format(
statement
)
)
for statement in sql_list
]
for i in range(1, len(tasks)):
tasks[i].set_upstream(tasks[i - 1])
flow.run()
Kevin Kho
Will Milner
03/26/2021, 6:07 PM