Hi folks,We're seeing some issues with Prefect Cloud. We're actively investigating, and will update this channel as soon as we have an update.Edit: this issue appears to be resolved. Thank you for your patience.
Hi all, we're experiencing some intermittent disruptions in the Prefect Cloud API; you may see some disruptions in the Prefect Cloud UI, including some issues logging in and viewing flows. All data is intact but you may have some brief flow and task run downtime. We're working on getting service restored as soon as possible and will update the channel as more information becomes available. Feel free to each out to me if you have any questions or concerns.
As a follow up to my earlier message, we wanted to let everyone know what went wrong and what we're doing to prevent similar outages in the future.What happened:
During a routine API release at approximately 09:28 EST, one of our API nodes became out of sync with the others due to a hanging application of metadata. As a result, traffic served to the unhealthy node incorrectly received 400 errors on any requests containing reference to flow schedules. While investigating the issue, our Cloud engineers were forced to re-apply the metadata, leading to a complete service disruption that lasted approximately 9 minutes. In this window, all communication with the API was severed, which means runs that were scheduled during that time may have been late and runs happening at that time may have become zombies. The intermittent disruptions lasted from approximately 09:45 EST to 11:00 EST.What we're doing to respond:
One of the services that make up the Cloud architecture has a mechanism that we use when applying database migrations; this mechanism is one that we've outgrown, and was the source of the disruption. Moving forward, we're moving to a more scalable API metadata application (removing the mechanism altogether) and introducing safeguards to the process in the form of pre-production replica DBs.
Hey folks! Apologies for the late Friday announcement but we are super excited to announce the release of Prefect 0.11.0 / 0.11.1! The headline feature of this release is first class file-based caching semantics (similar to Make / Luigi) for efficient reuse of task results. We wrote up a blog post describing these changes and more here: https://medium.com/the-prefect-blog/prefect-0-11-0-improved-apis-and-targets-79fef9d90aecHappy Engineering!
Hey everyone! Just wanted to blast out that I’m hosting a zoom call today for Prefect users at 4pm EDT / 1 pm PDT. This is a casual chance to get connected with other Prefect users over voice and see what they’re up to!We will do a few polls to break the ice and see what everyone is using in Prefect, and I’ve set up some breakout rooms so we can split up into smaller groups over topics if need be. One breakout room will be for totally new Prefect users where I will do a little demo for 10 minutes, so if you are are new or just curious about Prefect, please do stop by! :marvin:
Hey everyone! We have a flood of features and announcements queued up, but a few have already slipped out. Cloud users, you’ll notice a new graph whenever you’re looking at a flow or group of flows. This live-updating visualization will show you recent history, states, durations, and scheduled runs, as well as letting you jump quickly to runs of interest. Most importantly, it’ll give you an “at-a-glance” look at the health of your system.It’s the first in a series of “higher-order” ways of working with your data that @nicholas has been spearheading in the UI, and frankly we think it’s awesome. Please give us feedback and suggestions - you’ll see this popping up more and more soon!
Hey Everyone! I’d like to ask for your help to make Prefect Cloud and Prefect Server even better. Would you take 4 minutes to fill out this feedback survey?https://forms.gle/GArq27SyxBd8v4MCAThank you!
Hi all! Just want to officially circle back on the “Developing on Prefect’s Vue.js UI” livestream from May — we had many technical difficulties day-of 😬 but between some heavy editing and reshooting some of the screencasts and audio, the same content is now “remixed” and ready to view here 😁:
In it we walk you through setting up your development environment for front-end development on Prefect, and then walk through the front-end code for a PR I made with @nicholas’s help adding the “API Status” tile on Server’s UI. Enjoy!