https://prefect.io
Join Slack
Hello, is there an ongoing outage with Prefect Cloud? Seems some of our flows just stopped getting s...
# ask-community
j
Hello, is there an ongoing outage with Prefect Cloud? Seems some of our flows just stopped getting scheduled? I just had to manually toggle the flow off and on again to get it to produce future (scheduled) flow runs. Flow ID: f274e98a-3dcd-4823-8de7-4b74cc27615a
👀 2
✅ 1
l
I'm getting the same error
a
could you both provide more info? I'm investigating and trying to replicate
l
I'm using CronClock:
Copy code
flow.schedule = Schedule(clocks=[clocks.CronClock("0,5,10,15,20,25,30,35,40,45,50,55 * * * *")])
When I toggle the flow on/off, it schedules the next 10 runs, but doesn't schedule the ones after that like it's supposed to
a
@jars and @Leonardus Chen could you provide more info on when you started to see those issues? did you see that no more runs than the next 10 were scheduled and you had to toggle the schedule off and on again to get them back?
l
I noticed it a few hours ago but it may have started way earlier
👍 1
I can reproduce the issue with a brand new project and flow
Copy code
import prefect
from prefect import Flow, task
from prefect.schedules import Schedule, clocks


@task
def say_hello() -> None:
    logger = prefect.context.get("logger")
    logger.warning("Hello!")


if __name__ == "__main__":
    with Flow("hello-world") as flow:
        say_hello()

    flow.schedule = Schedule(clocks=[clocks.CronClock("* * * * *")])
    flow.register("hello-world")
Flow ID: 23909691-c46a-41d1-9624-ac93e9b76e77
So I registered the flow and it automatically schedules the next 10 runs: 00:24 - 00:33. But no further runs are scheduled beyond this point no matter how long I wait.
👍 1
a
thanks a lot
j
It looks like scheduled runs stopped getting queued around 1:00am Pacific this morning. The issue is identical to one encountered by many folks (including us) about 2 weeks ago... where Prefect Cloud would schedule the next 10 flow runs on toggling on/off -- but fail to continue scheduling automatically thereafter.
a
thanks for more info
j
I suspect many users of Prefect Cloud are running daily or hourly runs, so they would not have experienced the outage until 10 days or 10 hours respectively. But our flow executes every 10 minutes, so it only represents 1 hour & 40 minutes. So we're quick to realize when our PagerDuty alarm goes off.
👍 1
a
short update: we've identified the issue and new scheduled runs should be getting stable again within the next couple of minutes thanks so much for reporting the issue!
🙌 1
j
thakn you
@Anna Geller, is there a more appropriate place to file an operational issue that could lead to faster resolution times?
a
I think reporting this via Slack is the fastest, but for production incidents feel free to tag me directly and I will make sure to communicate with the team
you can even send me an email anna@prefect.io in case I don't see Slack
j
appreciate it @Anna Geller
🙌 1
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by