Madison Schott

    Madison Schott

    2 months ago
    Hi! My pipeline was successful except for this dbt freshness test so the result flow was green. Usually I receive a notification if a task fails and if the pipeline run is successful and found no alerts. Any ideas on why I wouldn't be receiving either of those?
    for example this is what one of the success messages used to look like
    Kevin Kho

    Kevin Kho

    2 months ago
    Maybe your alerts are on the Flow level and not task level?
    Madison Schott

    Madison Schott

    2 months ago
    How do I check that?
    Kevin Kho

    Kevin Kho

    2 months ago
    Are you using state handlers or Automations?
    Madison Schott

    Madison Schott

    2 months ago
    State handlers I believe
    Kevin Kho

    Kevin Kho

    2 months ago
    Can I see how you attached it?
    Madison Schott

    Madison Schott

    2 months ago
    I used the UI to set it up
    Kevin Kho

    Kevin Kho

    2 months ago
    I think that is on the Flow level and not task so you’d need to fail the flow is this task fails for that automation to trigger
    Madison Schott

    Madison Schott

    2 months ago
    Do you know why the flow doesn't fail if that task fails?
    Is it because it has no dependencies?
    Kevin Kho

    Kevin Kho

    2 months ago
    So by default, I think the last task is a reference task, but you can set them also. So if something in the middle fails but the last tasks end successfully, it will be labeled successful
    Madison Schott

    Madison Schott

    2 months ago
    Got it- thanks! I’ll look into this today and let you know if I have any questions.
    In our case, we make a simple tweak at the end of our script to tell this flow to use the run_spark_job task as its reference task:
    
    flow.set_reference_tasks([submitted])
    is this
    [submitted]
    in the docs supposed to be the
    run_spark_job
    ?
    oh wait never mind I think I see what the docs are referring to now, nvm!
    Kevin Kho

    Kevin Kho

    2 months ago
    Sounds good!
    Madison Schott

    Madison Schott

    2 months ago
    is it possible to set that to all_successful with the exception of one task?
    Kevin Kho

    Kevin Kho

    2 months ago
    I don’t think there is an easy way for that other than using python to manipulate a list of your tasks