Hey James! Yes, this is the guiding document to convincing my team that Prefect > Airflow 😆 , but more so looking for a speed analysis of similar flows.
Do you know if anyone has performed such an analysis?
2 years ago
I’m not aware of anyone who has done a proper speed comparison, but that is most likely because the architecture differences between Airflow and Prefect are so large that it doesn’t seem like a dimension that needs to be tested empirically. Airflow tasks take a minimum of 10 seconds to be scheduled and run, so two sequential tasks will take a minimum of 20 seconds just to be queued for execution. Prefect, on the other hand, does not bog the scheduler down with these details and can consequently execute sequential tasks in ~100 milliseconds. Similar comparisons can be made for parallel pipelines.