https://prefect.io logo
Title
c

Claire Scott

04/27/2023, 1:18 AM
Has anyone else had any issues with the scheduler in prefect cloud this-morning? mine stopped generating new scheduled flow runs, so just quietly stopped running when it ran out of existing yellow dots . turning the schedule on the deployment off and on again re-generated a new set, but no more where added, so a couple of hours later, it just stopped running again, just had to turn the deployments off and on for the third time today
πŸ‘€ 1
1️⃣ 1
g

gojaeheon

04/27/2023, 1:52 AM
Our service does, too. Is there any problem in scheduler~? @Prefect
c

Claire Scott

04/27/2023, 2:21 AM
thanks @gojaeheon for the +1 I just had to restart the deployments again, 4th time today,
g

gojaeheon

04/27/2023, 2:23 AM
Me too,, 4th time today.. We should restart the deployments all day..πŸ˜‚
e

Erik Amundson

04/27/2023, 2:35 AM
Having this same issue as well.
πŸ‘ 1
n

Nate

04/27/2023, 3:20 AM
hi all, sorry about the trouble, I will raise this - do you mind sharing whether you're on prefect cloud 1 or 2? i assume 2, just want to be sure
:thank_you: 2
c

Claire Scott

04/27/2023, 3:21 AM
Prefect cloud 2
:thank-you: 1
e

Erik Amundson

04/27/2023, 3:21 AM
Cloud 2
c

Claire Scott

04/27/2023, 3:21 AM
I'll dm you my workspace if it helps
g

gojaeheon

04/27/2023, 3:31 AM
cloud 2
n

Nate

04/27/2023, 3:38 AM
thank you all - I've raised internally and will have an update as soon as we can
πŸ™Œ 2
a

Alex Prejean

04/27/2023, 3:38 AM
we're having a similar problem with cloud 2
d

Dhruv

04/27/2023, 3:39 AM
same
n

Nate

04/27/2023, 4:19 AM
hi all - we've identified an issue with the scheduler and have pushed a fix flow run scheduling should now resume as normal - please let us know if your issue persists
πŸ™Œ 4
:thank_you: 2
:thank-you: 1
c

Claire Scott

04/27/2023, 4:22 AM
Looks promising so far, thanks Nate
t

Taran Singh

04/27/2023, 4:35 AM
@Nate Thank you
g

gojaeheon

04/27/2023, 5:18 AM
@Nate It doesn't seem to work again.. Are the others okay? @Claire Scott
c

Claire Scott

04/27/2023, 5:40 AM
I'm project broken again too @Prefect 😞
m

Mateo Merlo

04/27/2023, 1:45 PM
I’m having the same issue with the ECS agent and the deployments.
πŸ‘€ 1
n

Nate

04/27/2023, 2:06 PM
hi @Mateo Merlo - can you confirm you are still noticing issues with flow run scheduling now? could you describe what you’re seeing?
m

Mateo Merlo

04/27/2023, 3:40 PM
Hi @Nate , we have re deployed the agent and now seems that flows are working. Thanks!!
:thank-you: 1
n

Nate

04/27/2023, 3:47 PM
thanks for the update!
m

Mateo Merlo

05/03/2023, 6:01 AM
Hi @Nate, I'm having an issue that I think can be related with this previous one. At least 2 3 times a day we have a pipeline that crashed (Is not always the same pipeline but the error is the same. Image). Every time that a pipeline crashed, the ECS task keeps running indefinitely (we have to stop it manually) and the agent (running in a different ECS task) is not able to continue with other pipelines so I ends with a lot of Late pipelines.