Seeing a new pattern today running tasks on ECS The prefect

Question

Seeing a new pattern today running tasks on ECS The prefect logs show `Flow run SUCCESS all reference tasks succeeded` and then the next log line says `No heartbeat detected from the remote task marking the run as failed `

jack · Answer

Here is a screenshot of the logs

jack · Answer

The flow run id is `2bb7aa77 8f9f 4b56 933c a0c1a53636a7`

jack · Answer

Using prefect cloud The ECS agent says core version 1 2 1

jack · Answer

The ECS page shows a normal exit

Anna Geller · Answer

Are you on Prefect Cloud or Server You posted on Server and in that case flow run ID wouldn t help <https discourse prefect io t flow is failing with an error message no heartbeat detected from the remote task 79|This page> explains the issue and shows some things you may do

jack · Answer

On prefect cloud Which is the best channel to post in

Anna Geller · Answer

< CL09KU1K7|prefect community> for the next time slightly smiling face thx I ll check the logs then Have just 10 min then I need to run

Anna Geller · Answer

do you know which task caused that issue

Anna Geller · Answer

do you have any long running jobs

jack · Answer

This task usually runs in 5 or 10 minutes

jack · Answer

The flow only has one task

Anna Geller · Answer

yup you do just some dataframe manipulation really weird did rerunning the flow help

Anna Geller · Answer

maybe some timeout or network connectivity issue on ECS side hard to determine any issue from logs it looks like all the work has been done successfully only not marked as success correct

jack · Answer

We kick off about 30 flow runs all the same flow We kicked off the batch three times and on the third time all flow runs succeeded

jack · Answer

Yes it appears to be a success but then gets marked as failure Makes me question race condition

Anna Geller · Answer

good thinking could be

jack · Answer

ECS shows the task as excited normally

Anna Geller · Answer

in that case perhaps you could add an Automation on this flow you could say if the flow run doesn t finish within X time or if it doesn t finish successfully trigger a new flow run

Anna Geller · Answer

you could set it up from the UI might be useful at least as a temporary solution over the weekend sweat smile

Anna Geller · Answer

well the container didn t throw any non zero exit code so from ECS perspective it s all fine as long as the container doesn t crash