Adrian Brudaru

01/20/2023, 9:03 AM
Hey guys, I'm a long time airflow user and considering prefect. How should I think about prefect? what are the main use cases I should have in mind when choosing Prefect over airflow?
wondering if it helps data scientists more for example, over airflow - is it easier to use? or what benefits does it bring to a team to use prefect over airflow?
I read some posts about it and they describe it rather being preference and some convenience
I'm wondering about the impact on the team outcomes, rather than preferences

Peyton Runyan

01/20/2023, 1:32 PM
I don't have any links to case studies on hand, but the biggest thing in terms of outcomes is speed of development and increasing the number of people who are able to contribute. Prefect 2 is essentially straight python with very few framework-specific constructs when writing the actual pipelines. If you can reason in python and use python's control flow constructs (loops, if/else, etc.), you can write flows in Prefect 2. Airflow requires knowing both airflow and whatever constructs your org has built around airflow to make it handle production use cases. That means it takes a while to get people up to speed. Or you have to have somebody who knows both airflow and the supplied pipeline code well enough to translate it to airflow.
:gratitude-thank-you: 1
Before joining Prefect I ran a small data science team, and we were able to take notebooks from analysts with a bunch of procedural code, and get them running on Prefect in just a few minutes after moving the code into basic python functions and adding a couple of prefect-specific touch. This was with Prefect 1. With Prefect 2, you also have regular python control flow, so there's very few prefect-specific touches besides the decorators for flows and tasks that you'll be dealing with.

Taylor Curran

01/20/2023, 2:53 PM
Hi Adrian, Airflow has improved their syntax a bit and now it is pretty similar to Prefect with decorator-like syntax. The similarity in how work-flows are defined is deceptive though. Although an Airflow script and a Prefect script look pretty similar, with Airflow you’re still unable to run the flows locally by just calling the python function. In prefect you can call the python function to run flows and tasks locally, no boiler-plate code required. This feature of Prefect 2 enables a few key user stories: • You can test your flows and tasks in the same way that you test python functions. • You can step through flows and tasks like you would step through python functions — so using your fave debugger is possible. • Beyond dev and testing, this feature enables you to execute flows in a serverless fashion without the requirement of a Prefect agent.
:gratitude-thank-you: 1

Anna Geller

01/20/2023, 2:53 PM
what are the main use cases I should have in mind when choosing Prefect over airflow?
I'm not sure whether you are asking yourself the right questions. what problem are you trying to solve? what are you trying to accomplish?
:upvote: 1
other helpful questions: • what are the skills in your team? do they already know Airflow? do they know Python and if so how well? • what constraints do you have in terms of resources? • what are your privacy requirements? • where do you want your code to run e.g. public cloud, on-prem, serverless, K8s?