https://prefect.io logo
y

Yaron Levi

07/19/2023, 2:11 PM
Hi 👋 A Run is late for 10 hours for no reason. Using Prefect Cloud. Adding more details in a thread. Would love to get help on the issue or a point of direction for investigation.
1
CleanShot 2023-07-19 at 17.11.42@2x.jpg
The backing infra is ECS tasks, but I don’t think it’s related in this case, since an ECS task wasn’t even started.
Normally, when there are problems in the ECS Task side of things, the Run is in Pending, and the AWS fails to start an ECS Task for some reason.
Here, it’s as if the Prefect Cloud itself just missing this Run and haven’t started any action from it’s side.
The late run Id:
j

Jake Kaplan

07/19/2023, 2:36 PM
hey, thanks for flagging this. will take a look
y

Yaron Levi

07/19/2023, 2:38 PM
@Jake Kaplan Thanks. If you need more info to help the debug, please ask.
e

Emil Christensen

07/19/2023, 2:57 PM
@Yaron Levi Jut to confirm... the run never transitioned to
Late
, and the work queue is currently healthy?
j

Jake Kaplan

07/19/2023, 3:30 PM
Hey @Yaron Levi if it's not a hassle could list the flow run ids you see on that page that appear stuck in scheduled?
y

Yaron Levi

07/19/2023, 3:33 PM
@Jake Kaplan Yes, It was stuck on Scheduled all the time (still is) and the work queue is healthy.
One second, I’ll bring the list.
Ah, ok, so there other ones are scheduled for the future and everything is ok.
I am only talking about the single one that is stuck (marked with red border in the image I’ve sent).
j

Jake Kaplan

07/19/2023, 3:35 PM
ahhh got it, thank you
👍 1
y

Yaron Levi

07/19/2023, 3:47 PM
@Jake Kaplan Do you want me to open an issue in Git for this?
j

Jake Kaplan

07/19/2023, 3:50 PM
no need atm but thank you for the offer, I think I have found the issue on our end. A very small number of flow runs appear have gotten stuck as you described Can you let me know though if you see your run change from
SCHEDULED
to either
LATE
or
PENDING
? I believe I resolved the issue would be good to know for sure
y

Yaron Levi

07/19/2023, 4:03 PM
OK, great it run now, thanks!
j

Jake Kaplan

07/19/2023, 4:11 PM
appreciate you reporting it 🙏
👍 1