Prefect’s unique value proposition - DAG visualizer, clean, easy to onboard and easy to integrate and deploy With so many tools for: 1. Task scheduling (e.g. Celery, Ray, dask, bullmq) and 2. Workflow/DAG scheduling (e.g. Apache Airflow and Prefect), one can get real puzzled on the workloads / use cases / applications. It’s hard to put Prefect on the map. Personally, it took me so much time to understand that Prefect is most suitable for data pipelines (e.g. ETLs, Data Science / ML pipelines) due to its: 1. DAG Visualization 2. Fine-grained per task configuration (a node in the DAG) by using decorators. While you could chain tasks using Celery and its web UI, it doesn’t come close to Prefect. 3. Development and production integration a. The most notable feature of Prefect is its integration (not really an integration per se but ability to easily get the YAML is something) to

docker compose

and to production environments, i.e. Kubernetes. b. Integrating the Helm Chart of Prefect was a breeze and we have it in production. c. As well as its ability to scale (we don’t use that currently). @justabill do you have collaterals for Prefect that shows where Prefect is in comparison to other/similar task/workflow schedulers?

Hi Ofir, thanks for laying our your use case so clearly. You could apply Prefect to implement a SAGA pattern over a microservice architecture, just as you could use step functions in AWS' example. As you noted, however, Prefect is more commonly used in data and machine learning engineering use cases, so there is not the same body of content around such use cases. Prefect doesn't maintain any content that make direct comparison's t alternative tools, but we have contributed to content to moderndatastack.xyz/, which offers summaries of alternatives.

Thanks bill I appreciate that! will definitely keep an eye on The Modern Data Stack. It’s very useful to be able to know what tools fit where