https://prefect.io logo
Title
m

Madison Schott

07/20/2022, 3:34 PM
Hi! My pipeline was successful except for this dbt freshness test so the result flow was green. Usually I receive a notification if a task fails and if the pipeline run is successful and found no alerts. Any ideas on why I wouldn't be receiving either of those?
for example this is what one of the success messages used to look like
k

Kevin Kho

07/20/2022, 4:15 PM
Maybe your alerts are on the Flow level and not task level?
m

Madison Schott

07/20/2022, 4:18 PM
How do I check that?
k

Kevin Kho

07/20/2022, 4:18 PM
Are you using state handlers or Automations?
m

Madison Schott

07/20/2022, 4:20 PM
State handlers I believe
k

Kevin Kho

07/20/2022, 4:23 PM
Can I see how you attached it?
m

Madison Schott

07/20/2022, 5:03 PM
I used the UI to set it up
k

Kevin Kho

07/20/2022, 5:05 PM
I think that is on the Flow level and not task so you’d need to fail the flow is this task fails for that automation to trigger
m

Madison Schott

07/20/2022, 5:14 PM
Do you know why the flow doesn't fail if that task fails?
Is it because it has no dependencies?
k

Kevin Kho

07/20/2022, 5:17 PM
So by default, I think the last task is a reference task, but you can set them also. So if something in the middle fails but the last tasks end successfully, it will be labeled successful
m

Madison Schott

07/21/2022, 1:59 PM
Got it- thanks! I’ll look into this today and let you know if I have any questions.
In our case, we make a simple tweak at the end of our script to tell this flow to use the run_spark_job task as its reference task:

flow.set_reference_tasks([submitted])
is this
[submitted]
in the docs supposed to be the
run_spark_job
?
oh wait never mind I think I see what the docs are referring to now, nvm!
k

Kevin Kho

07/21/2022, 8:49 PM
Sounds good!
m

Madison Schott

07/21/2022, 8:51 PM
is it possible to set that to all_successful with the exception of one task?
k

Kevin Kho

07/21/2022, 8:53 PM
I don’t think there is an easy way for that other than using python to manipulate a list of your tasks