Igor Morgunov
09/06/2022, 4:57 PM@task()
def task1():
# do stuff
@task()
def task2():
# do other stuff
ids = ['aaa', 'bbb', 'ccc']
for id in ids:
x = task1()
y = task2(upstream_tasks=[x])
flow.run(executor=LocalDaskExecutor)
My problem is that task2()
fires when the first instance of task1()
completes successfully - I need task2()
to fire only once all of task1()
instances have completed - what am I doing wrong here?Barada Sahu
09/06/2022, 5:08 PMwait_for=[x]
Nate
09/06/2022, 6:38 PMtask1.map(ids)
instead of a for loop and you need to define your flow as a `with Flow() as flow:`context manager
if prefect 2, there is no longer a concept of an Executor
, you'd want to use the DaskTaskRunner
wait_for
is indeed what you want if prefect 2, and upstream_tasks
is what you want for prefect 1.Igor Morgunov
09/07/2022, 8:40 AM