https://prefect.io logo
Title
g

Gosia

04/26/2023, 3:37 PM
Hi there. I’m evaluating Prefect for our ETL jobs that are getting increasingly complex. I have a group of tasks that can run independently of each other. After they’re all finished, I’d like to run another task, even if any of the tasks in the group has failed. What is the recommended way of achieving this effect in Prefect?
z

Zanie

04/26/2023, 3:40 PM
Yeah just run
final_task.submit(wait_for=[allow_failure(t) for t in upstream_futures])
g

Gosia

04/26/2023, 3:40 PM
Cool. I’m going to try it now. Thanks.
z

Zanie

04/26/2023, 3:41 PM
Where upstream futures is
upstream_futures = []
upstream_futures.append(indepent_task_1.submit(...))
upstream_futures.append(indepent_task_2.submit(...))
g

Gosia

04/26/2023, 3:42 PM
Yup. I’ve got this part already. Didn’t know about the
allow_failure
bit. Thanks!