Thread
#prefect-community
    Constantino Schillebeeckx

    Constantino Schillebeeckx

    5 months ago
    I have a flow that runs 121 mapped tasks - two of those tasks failed however the overall flow was nor marked as failed
    Both tasks failed because:
    No heartbeat detected from the remote task; marking the run as failed.
    Kevin Kho

    Kevin Kho

    5 months ago
    So for the heartbeat, it might be more stable with threads. Have you done that? With the flow marked as failed, are these mapped tasks the last tasks in your flow? The final state is determined by the
    terminal_tasks
    so you can explicit make them terminal tasks by doing:
    flow.terminal_tasks = [...]
    Constantino Schillebeeckx

    Constantino Schillebeeckx

    5 months ago
    no, these tasks run in the middle of the flow
    in general though, even if the task fails, shouldn't the flow be marked as failed?
    Kevin Kho

    Kevin Kho

    5 months ago
    not really unless you explicitly specify. for example if you have a flow with A->B->C and B fails but you force C to run and it’s successful, I think this is considered a success by default because C is the terminal task in the graph
    normally failure would propagate and fail the last task as well but in the event it doesn’t, you have to be explicit about what task makes the flow fail
    Constantino Schillebeeckx

    Constantino Schillebeeckx

    5 months ago
    i see - i noticed in this particular case, the failed tasks didn't report failure until the all other tasks had run in the flow (which confuses me) - so technically the terminal task was a success
    to be a bit more explicit, i have mapped tasks
    L1
    ,
    Ln
    and then I have a task
    do_dbt
    which is a downstream dependencies of all the mapped tasks
    L
    - one of those mapped tasks stopped reporting a heartbeat back and eventually
    do_dbt
    just started running and passed. then
    Ln
    was finally set as failed
    Kevin Kho

    Kevin Kho

    5 months ago
    it sounds like the dependency was not set right. how did you make
    t
    a downstream?
    Constantino Schillebeeckx

    Constantino Schillebeeckx

    5 months ago
    Kevin Kho

    Kevin Kho

    5 months ago
    That looks perfectly right though. I would try setting
    flow.terminal_tasks = [L]
    but I’m not 100% sure on the mapped task behavior
    Constantino Schillebeeckx

    Constantino Schillebeeckx

    5 months ago
    but
    L
    aren't terminal tasks, and
    do-dbt
    should only run after all of
    L
    is finished
    is it bad form to set all tasks as terminal tasks to ensure this behavior doesn't happen?
    Kevin Kho

    Kevin Kho

    5 months ago
    My bad not terminal task
    flow.reference_tasks=[…]
    Constantino Schillebeeckx

    Constantino Schillebeeckx

    5 months ago
    i see - that does seem like a valid workaround, i'm curious though why
    do_dbt
    would run if one of the mapped
    L
    tasks has a missing heartbeat
    Kevin Kho

    Kevin Kho

    5 months ago
    Yeah I am not sure about that at the moment either
    Constantino Schillebeeckx

    Constantino Schillebeeckx

    5 months ago
    I've just bumped up my container resources and it looks like i'm not getting heartbeat issues anymore (thank you docs!) - i'm wondering if there's a bug for up/down stream task dependencies when this heartbeat issue comes up
    Kevin Kho

    Kevin Kho

    5 months ago
    Potentially, I can see that.