Hey guys! I'm new to Prefect as I'm trying new tools. How does Prefect compare to Dagster? Also, would MLFlow complement the use of Prefect?

Hey Jean, I’ll speak to the MLFlow section of this. I used MLFlow as a way to track experiment results on Databricks. The relationship would be wrapping that MLFlow writing as a Task and monitoring the success of writing out to MLFlow. MLFlow is also good for storing artifacts from model runs (plots, metrics).

I’ll add this article to your discovery gathering - it goes through the tools you mentioned in a pretty holistic way (since I may be biased to prefect duck).

@Kevin Kho thanks! And does it work alongside prefect?

I haven’t tried it, but they should work together without a problem because Prefect works with Python code and MLFlow has a Python API

Dagster and Prefect are actually quite different - Dagster’s focus appears to be on tracking data assets and providing things like type systems, etc.  They are very interested in what your tasks are doing.  This means that writing a Dagster pipeline requires thinking through config, and restructuring your code to fit their models. Prefect, on the other hand, is focused on the orchestration of your tasks and pipelines — things like scheduling, recovery from failure, running workflows in heterogeneous environments.  This means that writing Prefect pipelines requires only a light touch to your existing code.

Hey @Dylan thank you for the prompt response. Prefect does seem more straightforward. I'm def leaning more towards it, just gotta see how my pieline is gonna work with Prefect + MLFLlow

Let us know how we can help! 😄

What will I be missing from Prefect by not taking advantage of Prefect-Cloud other than the cloud part itself? Are there any features not in the open sourced part? I'm a bit confused on that

Hey @Amber Papillon, Take a look at the differences between Prefect Server and Prefect Cloud here. There are a few more differences between the two backend APIs that I can link as well (they seem to have moved…).

@Kyle Moon-Wright Thank you! Yes, I would appreciate it if you could. I didn't really understand the section "Scale & Performance" it's not very clear on the reason for that difference.

Hmm, it looks like the page I was referring to was removed for the time being as we are transitioning to success-based pricing, so I’ll need to check internally with the team for it’s whereabouts tomorrow. It was really just a feature list and doesn’t address your question ^. Did you have a question on the “Scale & Performance” section that I can clarify? Prefect Server was not intended for production purposes, but rather as a backend API for single developers/hobbyists to get started orchestrating with Prefect. Going the Server route, you are in charge of maintaining the networking between services and health of your containers - while Cloud was built with a highly-available and responsive API meant for production-level flows, taking care of the backend entirely so you can focus on your business critical work.

I do believe the Server will meet my needs! Thank you for the info!

Do you plan to use Spark and Databricks clusters?

I plan to use Spark/Vaex and occasionally Dask but that's unlikely. Databricks will be much later on.

Ok so MLFlow has a couple of features (Tracking, Deployment, Model Registry). I personally only have experience with Tracking. Tracking works by having a snippet of code where you log model hyperparameters and metrics and artifacts (plots or csvs). You would insert it in your code after model training. This is agnostic to Spark or Vaex or Pandas.

Yes, that does the trick. Thank you very much @Kevin Kho