Hi All,
The prefect tasks which are in resume stat...
# prefect-servers
Saurabh Indoria
03/08/2022, 8:11 AMHi All,
The prefect tasks which are in resume state are not going through, they remain in resume state. This is affecting our production flow runs as well. Can someone please check if there's any issue with prefect cloud?
cc: @Kevin Kho @Christina Lopez @Anna Geller
k
Kevin Kho
03/08/2022, 2:22 PM
Hey @Saurabh Indoria, there is nothing wrong with Prefect Cloud since you posted. Is this issue still persisting?
s
Saurabh Indoria
03/08/2022, 2:38 PMYes, this is still persisting
Saurabh Indoria
03/08/2022, 2:39 PMI can send you the flow run ID if that helps?
Saurabh Indoria
03/08/2022, 2:39 PMAll our production flows are stuck
k
Kevin Kho
03/08/2022, 2:40 PM
I could look yep. Are your agents healthy?
Kevin Kho
03/08/2022, 2:41 PM
And was this working before and then stopped working?
s
Saurabh Indoria
03/08/2022, 2:45 PMYes, we have been using the same flows since months now. Below are few examples..
https://cloud.prefect.io/quiltai/flow-run/b0d04920-1063-43db-848b-ebd93981a159
https://cloud.prefect.io/quiltai/flow-run/89b4a449-1c1a-4337-9cb3-d03f62cc89bf
https://cloud.prefect.io/quiltai/flow-run/8514e320-84a6-4e29-aa6c-285b01dcc929
Saurabh Indoria
03/08/2022, 2:46 PMAgents are healthy as well... all tasks run well, even the newly created ones... just the resume flow functionality is not working..
Saurabh Indoria
03/08/2022, 2:48 PMThis has happened for the second time now... couple of months back we saw the same issue which automatically resolved in 2 days without any official release from Prefect.. And now we are seeing this again and is a bit scary for us..
k
Kevin Kho
03/08/2022, 2:54 PM
Looking into it
s
Saurabh Indoria
03/08/2022, 2:54 PMThanks! Appreciate any help on this..
k
Kevin Kho
03/08/2022, 3:08 PM
There is another report of this so I think it’s something we need to look at. The team is already looking
🙌🏽 1
s
Saurabh Indoria
03/08/2022, 3:09 PMOkay, please keep me posted on it... thanks!
👍🏽 1
c
Christina Lopez
03/08/2022, 3:13 PMThanks for tagging me @Saurabh Indoria so I can be informed!
🙌 1
k
Kevin Kho
03/08/2022, 8:55 PM
Hey @Saurabh Indoria, we found an API query that was timing out in Cloud for the service that handles resuming these tasks. We made edits to it and deployed it and the new deployment just went through but it is still timing out so this is not resolved for now and we are still working on a fix. Sorry about that.
s
Saurabh Indoria
03/09/2022, 4:02 AMOkay.. Thanks for the update.. please keep me posted on further developments..
👍🏽 1
Saurabh Indoria
03/09/2022, 4:04 AM@Yash Joshi CC
k
Kevin Kho
03/09/2022, 6:43 PM
This got fixed literally today. We rolled out two changes on the backend so stabilize this service so you should not have problems going forward. New flows will already work. The ones stuck in pending will resolve as the service works through the backlog
upvote 1
c
Christina Lopez
03/09/2022, 6:48 PM@Saurabh Indoria @Yash Joshi ^^^
🙌 1
s
Saurabh Indoria
03/10/2022, 5:46 AMOkay great! Thanks @Kevin Kho!
k
Kevin Kho
03/10/2022, 5:48 AM
Sorry about that!
Kevin Kho
03/10/2022, 5:51 AM
I just checked my flows that were stuck and they are still pending. Will talk to the team about that tom but the new ones were definitely working when I tested
s
Saurabh Indoria
03/10/2022, 5:54 AMSure, thanks! We also did a hotfix in our production system to move away from pause-resume feature until it is stable enough..
Saurabh Indoria
03/11/2022, 9:27 AMLooks like even new flows get stuck about 15% of the times on Resume state. @Kevin Kho / @Christina Lopez Please get this up on priority if possible..
k
Kevin Kho
03/11/2022, 2:42 PM
Hi @Saurabh Indoria, sorry about that will check with the team again
Kevin Kho
03/11/2022, 2:49 PM
Could you give us a new flow run id?
c
Christina Lopez
03/11/2022, 3:02 PM@Saurabh Indoria ^
s
Saurabh Indoria
03/12/2022, 3:50 AMLooks like most of them eventually went through, however, there certainly is a delay for few flows..
Saurabh Indoria
03/12/2022, 3:51 AMI couldn't find flow run IDs which are stuck now, but once we start testing again on Monday, I'll share the flow runs if they get stuck
k
Kevin Kho
03/12/2022, 4:05 AM
Yes please!
2 Views