@Marvin how to continue with flow if a task throws an exception

In Prefect, if a task throws an exception, the task will be marked as

Failed

and the flow will continue to execute the remaining tasks. The flow run will continue even if a task fails. You can find more details in the Prefect documentation here.

@Marvin that is not true. ERROR | Flow run 'phenomenal-macaw' - Finished in state Failed. The flow finished after first task failed

I apologize for the confusion earlier. In Prefect, the default behavior is to stop the flow run if a task fails. However, you can change this behavior by using the

skip_on_upstream_skip

parameter in your task decorator. This will allow the flow to continue even if a task fails. Here is an example:

from prefect import task, Flow

@task(skip_on_upstream_skip=False)
def fail_task():
    raise ValueError("I fail successfully")

@task
def success_task():
    print("I'm fail safe!")

with Flow("My Flow") as flow:
    fail_task()
    success_task()

flow.run()  # This will print "I'm fail safe!" even though fail_task() raises an exception

In this example,

success_task

will run even if

fail_task

fails because

skip_on_upstream_skip=False

in the

@task

decorator for

fail_task

. Please refer to the Prefect documentation for more details: Prefect Docs

@Marvin how about version 2

In Prefect 2.0, the behavior is similar. If a task fails, it will be marked as

Failed

and the flow run will continue with the remaining tasks. However, you can change this behavior by using the

skip_on_upstream_skip

parameter in your task decorator. This will allow the flow to continue even if a task fails. Here is an example:

from prefect import task, Flow

@task(skip_on_upstream_skip=False)
def fail_task():
    raise ValueError("I fail successfully")

@task
def success_task():
    print("I'm fail safe!")

with Flow("My Flow") as flow:
    fail_task()
    success_task()

flow.run()  # This will print "I'm fail safe!" even though fail_task() raises an exception

In this example,

success_task

will run even if

fail_task

fails because

skip_on_upstream_skip=False

in the

@task

decorator for

fail_task

. Please refer to the Prefect documentation for more details: Prefect Docs

@Marvin Unexpected argument(s) for skip_on_upstream_skip

I apologize for the confusion earlier. In Prefect 2.0, if a task fails, by default the flow run will stop. However, you can change this behavior by setting the

skip_on_failure

parameter to

True

in your task decorator. This will allow the flow to continue even if a task fails. Here is an example:

from prefect import task, Flow

@task(skip_on_failure=True)
def fail_task():
    raise ValueError("I fail successfully")

@task
def success_task():
    print("I'm fail safe!")

with Flow("My Flow") as flow:
    fail_task()
    success_task()

flow.run()  # This will print "I'm fail safe!" even though fail_task() raises an exception

In this example,

success_task

will run even if

fail_task

fails because

skip_on_failure=True

in the

@task

decorator for

fail_task

. Please refer to the Prefect documentation for more details: Prefect Docs

@Marvin how to use exception_to_failed_state

In Prefect, if you want to handle exceptions and continue with the flow, you can use the

return_state=True

parameter when calling a task. This will return the state of the task instead of its result. If the task fails, you can check if the state is failed and handle the exception accordingly. Here is an example:

from prefect import task, Flow

@task
def fail_task():
    raise ValueError("I fail successfully")

@task
def success_task():
    print("I'm fail safe!")

with Flow("My Flow") as flow:
    state = fail_task(return_state=True)
    if state.is_failed():
        print("Oh no! The task failed. Falling back to '1'.")
        result = 1
    else:
        result = state.result()
    success_task()

flow.run()  # This will print "Oh no! The task failed. Falling back to '1'." and "I'm fail safe!"

In this example,

success_task

will run even if

fail_task

fails because we handle the exception from

fail_task

and continue with the flow. Please refer to the Prefect documentation for more details: Prefect Docs

hi @Hen rik - ill get prefect 1 out of his brain soon you can use

return_state=True

@Marvin please find a usage example of

return_state=True