https://prefect.io logo
Join Slack
Powered by
I have a flow that runs 121 mapped tasks - two of ...
# prefect-community
c
I have a flow that runs 121 mapped tasks - two of those tasks failed however the overall flow was nor marked as failed
discourse 1
Both tasks failed because:
No heartbeat detected from the remote task; marking the run as failed.
k
So for the heartbeat, it might be more stable with threads. Have you done that? With the flow marked as failed, are these mapped tasks the last tasks in your flow? The final state is determined by the 
terminal_tasks
so you can explicit make them terminal tasks by doing:
Copy code
flow.terminal_tasks = [...]
c
no, these tasks run in the middle of the flow
in general though, even if the task fails, shouldn't the flow be marked as failed?
k
not really unless you explicitly specify. for example if you have a flow with A->B->C and B fails but you force C to run and it’s successful, I think this is considered a success by default because C is the terminal task in the graph
normally failure would propagate and fail the last task as well but in the event it doesn’t, you have to be explicit about what task makes the flow fail
c
i see - i noticed in this particular case, the failed tasks didn't report failure until the all other tasks had run in the flow (which confuses me) - so technically the terminal task was a success
to be a bit more explicit, i have mapped tasks 
L1
, 
Ln
and then I have a task 
do_dbt
which is a downstream dependencies of all the mapped tasks 
L
- one of those mapped tasks stopped reporting a heartbeat back and eventually 
do_dbt
just started running and passed. then 
Ln
was finally set as failed
k
it sounds like the dependency was not set right. how did you make 
t
a downstream?
c
k
That looks perfectly right though. I would try setting
Copy code
flow.terminal_tasks = [L]
but I’m not 100% sure on the mapped task behavior
c
but 
L
aren't terminal tasks, and 
do-dbt
should only run after all of 
L
is finished
is it bad form to set all tasks as terminal tasks to ensure this behavior doesn't happen?
k
My bad not terminal task
flow.reference_tasks=[…]
upvote 1
c
i see - that does seem like a valid workaround, i'm curious though why 
do_dbt
would run if one of the mapped 
L
tasks has a missing heartbeat
k
Yeah I am not sure about that at the moment either
c
I've just bumped up my container resources and it looks like i'm not getting heartbeat issues anymore (thank you docs!) - i'm wondering if there's a bug for up/down stream task dependencies when this heartbeat issue comes up
k
Potentially, I can see that.
6 Views
Open in Slack
PreviousNext
Powered by