Noah Holm

    Noah Holm

    7 months ago
    Hey all, we ran into an unexpected (to us) behaviour today when a flow was considered successful even though some of it’s tasks failed and I’m curious to hear about potential solutions. We have this pattern of tasks in a flow with dependencies according to the arrows:
    task1
    ->
    task2
    ->
    task3
    Task3 is the last task of the flow and has other upstream dependencies which might cause task3 to be skipped in a flow run. Today a flow run got the following states:
    Failed
    ->
    TriggerFailed
    ->
    Skipped
    I assume the flow is considered successful since task3 was skipped and skipped states are considered successful (that makes sense). But we don’t want to consider a flow run successful if any tasks in the flow has a failed state. Refreshing my skills in the docs I’m thinking of handling it with state handlers, but I assume that a state handler on the flow would still see the flow state as successful due to the last skipped task. Is the only way forward here to have a state handler on each of the tasks in the flow?
    Anna Geller

    Anna Geller

    7 months ago
    Close but no cigar! 🙂 instead of state handlers, you would need to use reference tasks. Here is how you can set it:
    flow.set_reference_tasks(
            [
                task1,
                task2,
                task3,
            ]
        )
    This way, if any of those 3 tasks fails, the flow run will be considered Failed as well.
    Noah Holm

    Noah Holm

    7 months ago
    So simple, so effective. Thanks Anna
    For history, this doc part explained what I needed to know https://docs.prefect.io/core/concepts/flows.html#reference-tasks