https://prefect.io logo
#prefect-community
Title
# prefect-community
c

Constantino Schillebeeckx

04/20/2022, 3:19 PM
I have a flow that runs 121 mapped tasks - two of those tasks failed however the overall flow was nor marked as failed
discourse 1
Both tasks failed because:
No heartbeat detected from the remote task; marking the run as failed.
k

Kevin Kho

04/20/2022, 3:23 PM
So for the heartbeat, it might be more stable with threads. Have you done that? With the flow marked as failed, are these mapped tasks the last tasks in your flow? The final state is determined by the
terminal_tasks
so you can explicit make them terminal tasks by doing:
Copy code
flow.terminal_tasks = [...]
c

Constantino Schillebeeckx

04/20/2022, 3:24 PM
no, these tasks run in the middle of the flow
in general though, even if the task fails, shouldn't the flow be marked as failed?
k

Kevin Kho

04/20/2022, 3:50 PM
not really unless you explicitly specify. for example if you have a flow with A->B->C and B fails but you force C to run and it’s successful, I think this is considered a success by default because C is the terminal task in the graph
normally failure would propagate and fail the last task as well but in the event it doesn’t, you have to be explicit about what task makes the flow fail
c

Constantino Schillebeeckx

04/20/2022, 3:54 PM
i see - i noticed in this particular case, the failed tasks didn't report failure until the all other tasks had run in the flow (which confuses me) - so technically the terminal task was a success
to be a bit more explicit, i have mapped tasks
L1
,
Ln
and then I have a task
do_dbt
which is a downstream dependencies of all the mapped tasks
L
- one of those mapped tasks stopped reporting a heartbeat back and eventually
do_dbt
just started running and passed. then
Ln
was finally set as failed
k

Kevin Kho

04/20/2022, 3:57 PM
it sounds like the dependency was not set right. how did you make
t
a downstream?
c

Constantino Schillebeeckx

04/20/2022, 3:57 PM
k

Kevin Kho

04/20/2022, 4:23 PM
That looks perfectly right though. I would try setting
Copy code
flow.terminal_tasks = [L]
but I’m not 100% sure on the mapped task behavior
c

Constantino Schillebeeckx

04/20/2022, 4:24 PM
but
L
aren't terminal tasks, and
do-dbt
should only run after all of
L
is finished
is it bad form to set all tasks as terminal tasks to ensure this behavior doesn't happen?
k

Kevin Kho

04/20/2022, 4:26 PM
My bad not terminal task
flow.reference_tasks=[…]
upvote 1
c

Constantino Schillebeeckx

04/20/2022, 4:28 PM
i see - that does seem like a valid workaround, i'm curious though why
do_dbt
would run if one of the mapped
L
tasks has a missing heartbeat
k

Kevin Kho

04/20/2022, 4:28 PM
Yeah I am not sure about that at the moment either
c

Constantino Schillebeeckx

04/20/2022, 4:44 PM
I've just bumped up my container resources and it looks like i'm not getting heartbeat issues anymore (thank you docs!) - i'm wondering if there's a bug for up/down stream task dependencies when this heartbeat issue comes up
k

Kevin Kho

04/20/2022, 4:45 PM
Potentially, I can see that.
6 Views