We’ll have full details rolling out soon, but Scheduler is really the fulfillment of one of our major goals: providing a free workflow management system that’s nonetheless backed by the same production infrastructure as Cloud. As a product, it’s aimed at individuals who don’t need (or want) the full Cloud platform, but who could definitely use some of Cloud’s features like a UI, scheduler, API, etc. to deploy their Core flows.

You’re teasing out all the cool details on a quiet Saturday afternoon 😉 Scheduler is tied to another thing we’ve been quiet about, honestly because we were getting some very interesting patents on it — we call it our hybrid execution engine. It allows you to keep code, data, and execution private while Prefect manages the orchestration. You can run your flows wherever you want — your laptop, kubernetes cluster, any cloud provider’s free tier, somewhere we host for you — we’re completely agnostic. It’s like “on-prem lite” for everyone, except you don’t have to do anything but kick off a Prefect agent from your CLI.

And we’ve been sneaking it all into Core over the last month or two, so we hope once we announce it, users will take up the baton and help Prefect flows run on any environment imaginable

But to your question — Scheduler will allow up to 5,000 successful task runs per month (of unlimited time, resources, and concurrency, thanks to the hybrid model). Failures don’t count against the total.

Actually another question on this! I’m assuming this follows the at least once processing model? Or is that more dependent on each individual executor more than the scheduler portion? The thing I’d want to make sure is that all execution options have the same processing guarantees so any kind of defensive code level checks don’t need to change. Edit: I know I at least remember reading up on how the Dask executor works and how if the scheduler dies, basically everything falls apart and can’t recover intelligently. I was just wondering if you solved that problem somehow or are accepting that’s a problem of using Dask as an execution engine.

This means that failures in execution CAN have adverse effects

because Cloud will not resubmit a run automatically that started and did not finish

(By the way, this doesn’t include things like manual or automatic retries — if you want a task to retry of course it will!)

We’re exploring allowing users to indicate that tasks can safely be rerun, but we’ve started with the safest/most conservative approach for most production workloads

cc @Chris White

This is a whole topic of Cloud we call “Zombies” — tasks that failed not because they entered a failed state, but because we detected that they aren’t actually alive

Ah, I was going to link you to an issue tracking this feature but it’s in our Cloud repo. Suffice to say it’s being evaluated!

but these are all excellent points for us to incorporate

Ah yeah I definitely like that more. Executing code on every failure would be a bit much. Anyway, I love what you guys are doing with Prefect in general!!

Just following up: https://medium.com/the-prefect-blog/a-scheduler-for-the-internet-fd706336e2fe