Seeing a new pattern today running tasks on ECS. T...
# prefect-serverj
jack
05/14/2022, 2:45 PMSeeing a new pattern today running tasks on ECS. The prefect logs show
Flow run SUCCESS: all reference tasks succeededNo heartbeat detected from the remote task; marking the run as failed.jack
05/14/2022, 2:46 PMHere is a screenshot of the logs
jack
05/14/2022, 2:47 PMThe flow-run-id is
2bb7aa77-8f9f-4b56-933c-a0c1a53636a7jack
05/14/2022, 2:49 PMUsing prefect cloud.
The ECS agent says "core version 1.2.1"
jack
05/14/2022, 2:58 PMThe ECS page shows a normal exit
a
Anna Geller
05/14/2022, 3:36 PM
Are you on Prefect Cloud or Server? You posted on Server and in that case flow run ID wouldn't help
This page explains the issue and shows some things you may do
j
jack
05/14/2022, 3:37 PMOn prefect cloud. Which is the best channel to post in?
a
Anna Geller
05/14/2022, 3:38 PM
#prefect-community for the next time 🙂 thx, I'll check the logs then. Have just 10 min then I need to run
Anna Geller
05/14/2022, 3:39 PM
do you know which task caused that issue?
Anna Geller
05/14/2022, 3:39 PM
do you have any long running jobs?
j
jack
05/14/2022, 3:39 PMThis task usually runs in 5 or 10 minutes
👍 1
jack
05/14/2022, 3:40 PMThe flow only has one task
a
Anna Geller
05/14/2022, 3:41 PM
yup, you do just some dataframe manipulation, really weird - did rerunning the flow help?
Anna Geller
05/14/2022, 3:42 PM
maybe some timeout or network connectivity issue on ECS side? hard to determine any issue from logs - it looks like all the work has been done successfully, only not marked as success, correct?
j
jack
05/14/2022, 3:43 PMWe kick off about 30 flow runs (all the same flow)
We kicked off the batch three times and on the third time all flow runs succeeded.
jack
05/14/2022, 3:44 PMYes, it appears to be a success but then gets marked as failure.
Makes me question race condition.
a
Anna Geller
05/14/2022, 3:44 PM
good thinking, could be
j
jack
05/14/2022, 3:44 PMECS shows the task as excited normally.
a
Anna Geller
05/14/2022, 3:45 PM
in that case, perhaps you could add an Automation on this flow? you could say: if the flow run doesn't finish within X time or if it doesn't finish successfully, trigger a new flow run
Anna Geller
05/14/2022, 3:45 PM
you could set it up from the UI - might be useful at least as a temporary solution over the weekend 😅
Anna Geller
05/14/2022, 3:47 PM
well, the container didn't throw any non-zero exit code so from ECS perspective it's all fine as long as the container doesn't crash
7 Views