https://prefect.io logo
Title
a

Andreas Nigg

04/04/2023, 7:11 AM
Hey folks, prefect 2.0 cloud user here. Yesterday evening (CET) prefect flows started to fail with ``Server error '500 Internal Server Error'`` and after runs stopped failing ALL deployments had a pending run - and the runs stayed in pending forever. They stayed in pending even though we have an automation which says "kill all runs staying in pending for longer than 1 hour". I can't provide any logs, as there were none, I guess this was related to the cloud outage described above. That's fine for me, however, I have a question related to that: Is there a way to prevent this "staying in pending forever"? We have some sensitive pipelines with concurrency 1 - so if one of the runs is in pending - all the others will wait forever.
w

Will Raphaelson

04/04/2023, 2:45 PM
Hey Andreas, i believe that the same service degradation that caused the 500s also caused the triggers to fail to evaluate. I dont think this should be a reoccurring issue. Have you experienced it before?
a

Andreas Nigg

04/04/2023, 2:58 PM
Hey, no I didn't find failed triggers from before. However I sometimes have flow runs which are stuck in pending for a long time. Is there anything you know about that? I can't provide logs, as there are none.
w

Will Raphaelson

04/04/2023, 3:07 PM
Hmm, hard to say without a little more info on the setup. want to chat for 15 or so and see if we can identify the bottleneck? Im flexible tomorrow from about noon to 3 central us time.
👀 1