https://prefect.io logo
Title
s

Stéphanie Cérulli

06/23/2022, 2:00 PM
hello, we have a problem with scheduled flows, since 2 PM French time, no more than 10 runs are running
k

Kevin Kho

06/23/2022, 2:02 PM
I don’t think this is expected in any way. Do you mean 10 runs for a given flow or 10 runs across the tenant?
s

Stéphanie Cérulli

06/23/2022, 2:03 PM
for a given flow
we have to disable and reactivate the schedule to have again 10 flows upcoming and after the 10 have ran it stops
k

Kevin Kho

06/23/2022, 2:08 PM
i think this has always been a limitation (10 flows specifically, and I can explain more). did you have different behavior before?
s

Stéphanie Cérulli

06/23/2022, 2:08 PM
yes a scheduled task running every 2 minutes should not stop after 10 runs
we have it for more than one year and it never stopped before today
k

Kevin Kho

06/23/2022, 2:12 PM
Ok so you find new ones are not being scheduled at all until you toggle?
s

Stéphanie Cérulli

06/23/2022, 2:12 PM
exactly
after 10 runs
k

Kevin Kho

06/23/2022, 2:13 PM
Did you re-register recently (no idea if relevant but just trying to collect more info)?
s

Stéphanie Cérulli

06/23/2022, 2:14 PM
no
we tried to restart agents, but it didn't change anything
k

Kevin Kho

06/23/2022, 2:16 PM
Can you DM me your tenant name and the flow id this happens for and I’ll need to ask some team members and get back to you
m

Mickael Riani

06/23/2022, 2:20 PM
what do you mean by tenant name ? the api_token ?
client = prefect.Client(api_token=tenant)
k

Kevin Kho

06/23/2022, 2:23 PM
The thing after the slash in the URL like this:
<https://cloud.prefect.io/kevin-prefect-io-s-account>
j

Jars

06/23/2022, 3:10 PM
We are also having this issue
Please see Flow ID d8d77c14-6aa8-401d-b3f3-69e3aea09cd7
It's on an indefinite 10-minutely schedule. We just deployed it at 6am Pacific, and it ran until now, and pending runs are drying up
k

Kevin Kho

06/23/2022, 3:13 PM
Yes found from the initial report that the scheduler has degraded performance. Currently being worked on to fix
j

Jars

06/23/2022, 3:20 PM
Thank you
Our scheduled runs are back now
k

Kevin Kho

06/23/2022, 3:41 PM
Umm…it’s still being worked on. If you toggled, you may get new runs
Fixed now. Thanks everyone!