Hey team, somehow today our agent is not picking up any flow runs to run. The last run happened around 24 hours ago, and since then it's gone quiet, I restarted it and it just sits there reporting

Waiting for flow runs...

We are cloud users, and we don't think we touched anything at all since it was last healthy (it was a bank holiday here yesterday in fact)

Hi @Kostas Chalikias - looks like there's a mismatch between the labels on your agent and the labels on your flow/flow runs. Can you check if there were any changes?

pretty sure there weren't, we noticed this in the release notes

Ok. Can you check what labels are on your agent and what labels are on one of the flow runs you want it to pick up?

will check & revert

Hi @Jenny, I've started an agent up locally to try and test and debug things. It's got labels

staging

(corresponding to our environment) and my hostname (from

<flow_name>.storage.labels

I believe) amongst a list of all the names of all the flows we have. The flow I'm trying to run has just the two labels (

staging

and my hostname). Nothing gets sent to the agent though when I hit run from the UI. It just says waiting for flow runs. I'm starting to suspect this isn't just labels. Wondered if you had any thoughts

Hmmm.... that sounds correct. Let me see what else what might be the issue.

Thanks Jenny

Hi @Freddie - a few questions to see if we can figure this out: • Is this happening with all your agents? Or just one of them? • Same q with flows - are all affected? Or just one? • What version of core are you running? Is the agent running with the same version? Thanks!

We've been running the latest 0.13.x and indeed still are on our production stack. We've deployed 0.14.x locally and on our staging instance. This is happening across all agents - we have one per stack, and then I'm running one locally too which mirrors staging.

@Jenny any other thoughts on this? Being without our scheduler is obviously not ideal

Hi @Kostas Chalikias - Looking into it right now to see if we can figure out what's going on.

Hi @Jenny - we've been triggering runs using the quick run and run page and they've been scheduling fine, but they never start, even with pressing the start now button. We just get an additional scheduled line on the timeline.

OK - I think I know the issue. Do you by any chance have more than 625 late runs? Our work queue pauses over that number and so no new runs will get picked up.

We've had a lot of late runs recently

Can you turn off the schedule on some of your flows to clear the number of late runs? (They can be toggled back on afterwards.)

I'll check our stack - we've both been clearing them relatively frequently

Thanks! Once they are cleared I'm hopeful that your new runs should get picked up. If they don't, please let me know and we'll keep investigating!

A couple have started running

Fantastic! Thanks for the update and your patience!

We'll have to keep an eye on things going forward and look at why we got to this level of late runs

Sounds good. Thank you!

Nice, thanks!