https://prefect.io logo
Title
m

Mitchell Bregman

07/31/2020, 6:37 PM
Hi there, has anyone done a performance analysis comparing Prefect to Airflow? Is there some sort of white paper I can use to present to my team?
j

james.lamb

07/31/2020, 6:46 PM
m

Mitchell Bregman

07/31/2020, 6:54 PM
Hey James! Yes, this is the guiding document to convincing my team that Prefect > Airflow 😆 , but more so looking for a speed analysis of similar flows.
Do you know if anyone has performed such an analysis?
c

Chris White

07/31/2020, 7:01 PM
I’m not aware of anyone who has done a proper speed comparison, but that is most likely because the architecture differences between Airflow and Prefect are so large that it doesn’t seem like a dimension that needs to be tested empirically.  Airflow tasks take a minimum of 10 seconds to be scheduled and run, so two sequential tasks will take a minimum of 20 seconds just to be queued for execution.  Prefect, on the other hand, does not bog the scheduler down with these details and can consequently execute sequential tasks in ~100 milliseconds.  Similar comparisons can be made for parallel pipelines.
:upvote: 1
m

Mitchell Bregman

07/31/2020, 7:11 PM
@Chris White copy and pasting this into my PPT deck 😉 Thank you so much for all you do!!! I ❤️ Prefect
c

Chris White

07/31/2020, 7:13 PM
no problem! Let us know how we can help 😄
m

Mitchell Bregman

07/31/2020, 7:15 PM
@curtis smiley ^^