https://prefect.io logo
Title
f

Farid

02/24/2023, 4:57 AM
Hi all, How can I handle the failure in task so that the whole flow doesn't fail because of that when it's being called in a mapped mode? For example, when a shell task is being called to run multiple commands in parallel using
.map
, the whole flow fails immediately if one of the child tasks fail. I have tried passing
return_state=True
but it doesn't seem be effective in mapped mode.
k

Khuyen Tran

02/24/2023, 4:27 PM
The whole flow shouldn’t fail immediately if one of your task fails. Which version of Prefect are you in? In the following example, the rest of the tasks still run when one task failed
from prefect import task, flow


@task
def task1(num: int):
    if num == 2:
        raise ValueError
    return num


@task
def task2(num: int):
    return num + 1


@flow
def mapped_flow():
    futures = task1.map(list(range(5)))
    return task2.map(futures)


mapped_flow()
Output:
10:27:36.103 | INFO    | Task run 'task1-0' - Finished in state Completed()
10:27:36.109 | INFO    | Task run 'task1-3' - Finished in state Completed()
10:27:36.224 | INFO    | Task run 'task1-1' - Finished in state Completed()
10:27:36.227 | INFO    | Task run 'task1-4' - Finished in state Completed()
10:27:36.456 | INFO    | Task run 'task2-0' - Finished in state Completed()
10:27:36.490 | ERROR   | Task run 'task1-2' - Finished in state Failed('Task run encountered an exception: ValueError\n')
10:27:36.581 | INFO    | Task run 'task2-4' - Finished in state Completed()
10:27:36.589 | INFO    | Task run 'task2-1' - Finished in state Completed()
10:27:36.624 | INFO    | Task run 'task2-3' - Finished in state Completed()
10:27:36.871 | ERROR   | Flow run 'tan-marmoset' - Finished in state Failed('1/5 states are not final.')