https://prefect.io logo
Title
j

jars

07/03/2022, 6:16 AM
Hello, is there an ongoing outage with Prefect Cloud? Seems some of our flows just stopped getting scheduled? I just had to manually toggle the flow off and on again to get it to produce future (scheduled) flow runs. Flow ID: f274e98a-3dcd-4823-8de7-4b74cc27615a
👀 2
1
l

Leonardus Chen

07/03/2022, 11:59 AM
I'm getting the same error
a

Anna Geller

07/03/2022, 12:00 PM
could you both provide more info? I'm investigating and trying to replicate
l

Leonardus Chen

07/03/2022, 12:01 PM
I'm using CronClock:
flow.schedule = Schedule(clocks=[clocks.CronClock("0,5,10,15,20,25,30,35,40,45,50,55 * * * *")])
When I toggle the flow on/off, it schedules the next 10 runs, but doesn't schedule the ones after that like it's supposed to
a

Anna Geller

07/03/2022, 12:18 PM
@jars and @Leonardus Chen could you provide more info on when you started to see those issues? did you see that no more runs than the next 10 were scheduled and you had to toggle the schedule off and on again to get them back?
l

Leonardus Chen

07/03/2022, 12:27 PM
I noticed it a few hours ago but it may have started way earlier
👍 1
I can reproduce the issue with a brand new project and flow
import prefect
from prefect import Flow, task
from prefect.schedules import Schedule, clocks


@task
def say_hello() -> None:
    logger = prefect.context.get("logger")
    logger.warning("Hello!")


if __name__ == "__main__":
    with Flow("hello-world") as flow:
        say_hello()

    flow.schedule = Schedule(clocks=[clocks.CronClock("* * * * *")])
    flow.register("hello-world")
Flow ID: 23909691-c46a-41d1-9624-ac93e9b76e77
So I registered the flow and it automatically schedules the next 10 runs: 00:24 - 00:33. But no further runs are scheduled beyond this point no matter how long I wait.
👍 1
a

Anna Geller

07/03/2022, 12:31 PM
thanks a lot
j

jars

07/03/2022, 1:04 PM
It looks like scheduled runs stopped getting queued around 1:00am Pacific this morning. The issue is identical to one encountered by many folks (including us) about 2 weeks ago... where Prefect Cloud would schedule the next 10 flow runs on toggling on/off -- but fail to continue scheduling automatically thereafter.
a

Anna Geller

07/03/2022, 1:05 PM
thanks for more info
j

jars

07/03/2022, 1:30 PM
I suspect many users of Prefect Cloud are running daily or hourly runs, so they would not have experienced the outage until 10 days or 10 hours respectively. But our flow executes every 10 minutes, so it only represents 1 hour & 40 minutes. So we're quick to realize when our PagerDuty alarm goes off.
👍 1
a

Anna Geller

07/03/2022, 1:30 PM
short update: we've identified the issue and new scheduled runs should be getting stable again within the next couple of minutes thanks so much for reporting the issue!
🙌 1
j

jars

07/03/2022, 1:30 PM
thakn you
@Anna Geller, is there a more appropriate place to file an operational issue that could lead to faster resolution times?
a

Anna Geller

07/03/2022, 1:33 PM
I think reporting this via Slack is the fastest, but for production incidents feel free to tag me directly and I will make sure to communicate with the team
you can even send me an email anna@prefect.io in case I don't see Slack
j

jars

07/03/2022, 2:19 PM
appreciate it @Anna Geller
🙌 1