https://prefect.io logo
Title
c

Constantino Schillebeeckx

04/29/2022, 2:24 PM
When I used
FilterTask
with my own filter func, does the task still filter out the default
NoResult
None
and exceptions?
When I execute the following flow it completes successfully even if there was an exception raised
foo = FilterTask(filter_func=lambda x: isinstance(x, int))


@task
def m1(x):
    if x == 5:
        raise ValueError()
    return x


with Flow('foo') as f:
    l1 = m1.map([1, 2, 3, 4, 5])
    foo(l1)
a

Anna Geller

04/29/2022, 3:13 PM
What is the use case you need this for?
My understanding of the filter task is to pass the mapped results to the downstream tasks. The flow run will be considered a success because foo filters out the failed tasks - does it make sense?
also, to change the behavior of when you would like the flow run to be considered a success, you can use:
flow.set_reference_tasks([your_important_task])
this way, regardless of the end result of the filterTask, you can customize when the flow run is considered Successful
c

Constantino Schillebeeckx

04/29/2022, 3:32 PM
my use case is that I want to know if a flow of mine fails; i use the filter task to filter our SKIP tasks, but i certainly still want to know if one of those tasks had an exception.
👍 1
The flow run will be considered a success because foo filters out the failed tasks - does it make sense?
that was my question, it seems like in addition to the
filter_func
the task will also filter out exceptions
the reference task doesn't work for me because i need to know whether any task in any part of the flow fails
a

Anna Geller

04/29/2022, 3:38 PM
I wonder whether a state handler might be easier to detect if some of those mapped tasks failed?
that was my question, it seems like in addition to the
filter_func
the task will also filter out exceptions
yup! you're right
the reference task doesn't work for me because i need to know whether any task in any part of the flow fails
Got it. I think state handler seems reasonable to figure that out and get notified. The only downside is: if you have 200 tasks and 20 of them fail, you'll get 20 notifications (which can get annoying), not a single one. If that's OK, then a state handler would be the best option
c

Constantino Schillebeeckx

04/29/2022, 3:49 PM
a state handler on the task or the flow?
a

Anna Geller

04/29/2022, 3:52 PM
on the mapped task to get notified about failure of all mapped child tasks
👍 1
c

Constantino Schillebeeckx

04/29/2022, 3:54 PM
thanks for all the clarifications! i couldn't do my job without you all ❤️
a

Anna Geller

04/29/2022, 3:55 PM
You're very welcome! ❤️