https://prefect.io
Join Slack
Hi! My pipeline was successful except for this dbt freshness test so the result flow was green. Usua...
# prefect-ui
m
Hi! My pipeline was successful except for this dbt freshness test so the result flow was green. Usually I receive a notification if a task fails and if the pipeline run is successful and found no alerts. Any ideas on why I wouldn't be receiving either of those?
for example this is what one of the success messages used to look like
k
Maybe your alerts are on the Flow level and not task level?
m
How do I check that?
k
Are you using state handlers or Automations?
m
State handlers I believe
k
Can I see how you attached it?
m
I used the UI to set it up
k
I think that is on the Flow level and not task so you’d need to fail the flow is this task fails for that automation to trigger
m
Do you know why the flow doesn't fail if that task fails?
Is it because it has no dependencies?
k
So by default, I think the last task is a reference task, but you can set them also. So if something in the middle fails but the last tasks end successfully, it will be labeled successful
m
Got it- thanks! I’ll look into this today and let you know if I have any questions.
Copy code
In our case, we make a simple tweak at the end of our script to tell this flow to use the run_spark_job task as its reference task:

flow.set_reference_tasks([submitted])
is this 
[submitted]
in the docs supposed to be the 
run_spark_job
?
oh wait never mind I think I see what the docs are referring to now, nvm!
k
Sounds good!
m
is it possible to set that to all_successful with the exception of one task?
k
I don’t think there is an easy way for that other than using python to manipulate a list of your tasks
9 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by