Prefect’s unique value proposition - DAG visualizer, clean, easy to onboard and easy to integrate and deploy
With so many tools for:
1. Task scheduling (e.g. Celery, Ray, dask, bullmq) and
2. Workflow/DAG scheduling (e.g. Apache Airflow and Prefect), one can get real puzzled on the workloads / use cases / applications.
It’s hard to put Prefect on the map.
Personally, it took me so much time to understand that Prefect is most suitable for data pipelines (e.g. ETLs, Data Science / ML pipelines) due to its:
1.
DAG Visualization
2.
Fine-grained per task configuration (a node in the DAG) by using decorators. While you could chain tasks using Celery and its web UI, it doesn’t come close to Prefect.
3.
Development and production integration
a. The most notable feature of Prefect is its integration (not really an integration per se but ability to easily get the YAML is something) to
docker compose
and to production environments, i.e.
Kubernetes.
b. Integrating the Helm Chart of Prefect was a breeze and we have it in production.
c. As well as its ability to scale (we don’t use that currently).
@justabill do you have collaterals for Prefect that shows where Prefect is in comparison to other/similar task/workflow schedulers?