Hey folks, prefect 2.0 cloud user here. Yesterday evening (CET) prefect flows started to fail with ``Server error '500 Internal Server Error'`` and after runs stopped failing ALL deployments had a pending run - and the runs stayed in pending forever. They stayed in pending even though we have an automation which says "kill all runs staying in pending for longer than 1 hour". I can't provide any logs, as there were none, I guess this was related to the cloud outage described above. That's fine for me, however, I have a question related to that: Is there a way to prevent this "staying in pending forever"? We have some sensitive pipelines with concurrency 1 - so if one of the runs is in pending - all the others will wait forever.

Hey Andreas, i believe that the same service degradation that caused the 500s also caused the triggers to fail to evaluate. I dont think this should be a reoccurring issue. Have you experienced it before?

Hey, no I didn't find failed triggers from before. However I sometimes have flow runs which are stuck in pending for a long time. Is there anything you know about that? I can't provide logs, as there are none.

Hmm, hard to say without a little more info on the setup. want to chat for 15 or so and see if we can identify the bottleneck? Im flexible tomorrow from about noon to 3 central us time.