https://prefect.io logo
#prefect-community
Title
# prefect-community
c

Christopher

03/27/2020, 8:35 AM
Hi folks, I lead an ML and data science team, and I’m evaluating a bunch of tools in this space. Prefect looks very nice indeed, but I am having trouble understanding exactly what I get with a paid plan vs the open source core. I have looked at all the feature lists I can find, but I guess I’m missing one of those classic “feature comparison tables” that one often sees. Can someone point me in the direction of something like that?
j

Jeremiah

03/27/2020, 12:05 PM
Hi @Christopher, next week we’re open-sourcing our server and UI which will blur the lines a bit from today, but gives us a good opportunity to re-clarify. In a nutshell: paid features include managed infrastructure (including auth, scale / HA, and security layers) as well as team management and permissions. Additional protective measures like services that enforce SLAs, attempt to restart flows that have stopped for unusual reasons are available. Platform also includes features that require global coordination of state, like concurrency limits for task, and third party integrations like Slack notifications.
a

Alf

03/27/2020, 12:16 PM
This belongs in #announcements
j

Jeremiah

03/27/2020, 12:18 PM
You know you’re right
j

Joe Schmid

03/27/2020, 1:41 PM
@Christopher We use Prefect Cloud (and used Core prior to that) for our data science workflows to train lots of ML models on healthcare data. If I can help answer any questions, feel free to DM me or just ask here. (Also wrote a high-level blog post about it late last year: https://www.symphonyrm.com/galaxy-brain-data-science-workflows-with-prefect-dask/ )
upvote 1
c

Christopher

03/27/2020, 9:26 PM
Thank you for the responses. 1. That announcement looks like great news 2. Thanks @Joe Schmid for the blog. I will read it and DM you if I have questions.
Hi again. Great blog post, thanks @Joe Schmid. @Jeremiah, most comparisons I see are between Prefect and Airflow - even Matt Rocklin makes this comparison. As a better alternative to airflow, I am already very convinced by Prefect’s value prop. What I am now unclear about is how Prefect compares or relates to something like Kubeflow. Is Prefect in a competing or complimentary space here? Also sorry if I come across as lazy and not wanting to do my own research - I promise this isn’t the case. I want to gear up my limited engineering resources to do the right side-by-side comparisons for our workflow and needs, and am just trying to get the best picture I can in advance. 😊
j

Jeremiah

03/29/2020, 2:29 AM
Hi @Christopher - don’t worry, it’s a natural comparison. In fact, I’m actually an Airflow PMC member. Prefect was explicitly designed to solve the issues with using Airflow in a modern PyData stack (or modern stack period, to be honest) — here’s our own take: https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 Typically, the difference between Prefect and most other workflow managers is fundamentally philosophical. For example, Prefect’s goal is to be workflow-agnostic and infrastructure-agnostic. That means our focus is on expressing a really powerful but lightweight syntax for defining workflows, and our challenge is making sure that can be applied in extremely general environments. We want Prefect to become the universal “glue” for whatever tools meet your needs. In contrast, a system like Kubeflow is much more opinionated: it’s for deploying machine learning on Kubernetes. If that’s your only purpose, then Kubeflow may have a more immediate impact, since your work already conforms to its assumptions. If that’s not how you already work, then you’re going to struggle with Kubeflow. At the end of the day, you should use the best tool for the job. If a more opinionated tool meets your needs, by all means use it! But if you’re experimenting, or you like having a local experience that’s identical to your deployed one, or you work in multiple environments or for multiple stakeholders, you may prefer Prefect’s more general Swiss-army-knife approach. We try to make it as easy as possible for you to dive in and find out!
👏 1
c

Christopher

03/29/2020, 2:32 AM
Thanks Jeremiah, really appreciate the reply. That gels with my own reading, so I feel like I have enough of a grip to get started on Monday. Really appreciate the engagement, thanks 😊
👍 1
j

Jeremiah

03/29/2020, 3:03 AM
Anytime! Hope you find Prefect useful 🙂