Prefect cloud scheduler bugged out and has schedul...
# ask-communitya
Austin Weisgrau
06/02/2023, 1:04 AMPrefect cloud scheduler bugged out and has scheduled dozens or hundreds of simultaneous scheduled runs for one of our deployments today
z
Zanie
06/02/2023, 1:31 AM
Which Prefect version are you using?
a
Austin Weisgrau
06/02/2023, 4:02 PMThat happened in prefect cloud, not a self run server
Our agent runs the latest version of the prefecthq/prefect:2-python3.10 docker image, as of yesterday, but this happened in the prefect cloud so the agent wouldn't be relevant right?
z
Zanie
06/02/2023, 4:03 PM
Generally. no. Thanks for the info — I was wondering if you were on Prefect 1.
Zanie
06/02/2023, 4:04 PM
Sorry you ran into an issue though. I’ll ask a Cloud engineer to reach out.
a
Austin Weisgrau
06/02/2023, 4:04 PMThanks !
z
Zanie
06/02/2023, 4:08 PM
Are you using automations to schedule runs?
a
Austin Weisgrau
06/02/2023, 4:09 PMNothing using the API or python triggers or stuff like that, we're using schedules on the deployments
Austin Weisgrau
06/02/2023, 4:10 PMThis was the first deployment I tried using an rrule schedule for, maybe I messed it up? Here's the rrule:
FREQ=WEEKLY;INTERVAL=1;BYDAY=FRAustin Weisgrau
06/02/2023, 6:54 PMIs there a way to limit concurrent runs of the same deployment to 1? This bug is quite problematic for a few reasons, one of which being that triggering dozens or hundreds of simultaneous runs is quite expensive on our execution infrastructure side
Austin Weisgrau
06/02/2023, 7:03 PMNVM found it - I can limit the work pool concurrency
Austin Weisgrau
06/02/2023, 7:04 PMShould I file a bug report on the git repo? This was a pretty dramatic bug and it's important that we get some followup that this won't happen again
z
Zanie
06/02/2023, 7:14 PM
You are welcome to open an issue — our on call engineer will be in touch with you regardless.
🙏 1
a
Andrew Brookins
06/02/2023, 7:29 PM👋 @Austin Weisgrau I’m on-call at the moment and will DM for more details.
d
Deceivious
06/05/2023, 8:02 AMJust out of curiosity, what was the root of the issue?
Deceivious
06/05/2023, 8:02 AMor the github issue link if some one has made one
a
Andrew Brookins
07/12/2023, 12:04 AMMy bad, I missed this reply! Yes, I can share details: https://github.com/PrefectHQ/prefect/pull/9872
2 Views