Hey folks, good morning (European time zone 🙂 ).
Due to yesterdays API degradations, some of my flows were way slower then normal, which resulted in them getting queued up. (I have a work queue with concurrency limit 1 due to the nature of the downstream system).
So flows started backing up and being late. So I had a flow which was running already 20 minutes which normally takes 2 minutes. As the good IT person I am I then just cancelled this flow. Cancellation did not work, so I deleted the flow - basically removing th bottleneck in the queue. But then, right after the bottleneck-flow-run was removed, ALL the late flows started to run, totally ignoring the concurrency limit of the queue.
(prefect 2.0 cloud, agents 2.10.11 on k8s)
Is this to be expected? Or should I open a github issue? And probably the better question: What additional information might I provide? Agent-Logs unfortunately only indicate normal operation.