https://prefect.io logo
#prefect-community
Title
# prefect-community
e

egk

04/14/2022, 6:57 PM
Is it a valid use-case to use Prefect for the sake of having a managed way of running tasks rather than "data pipelines"? I just want the workflow aspect of it.
k

Kevin Kho

04/14/2022, 6:58 PM
Hey I was happy to answer the Dagster question, but lost my three paragraphs when you deleted the message. We’re open to discussing other tools here. 😆. Typing a response.
e

egk

04/14/2022, 6:59 PM
Ohhh wow, I'm so sorry. Obviously I did not know that.
k

Kevin Kho

04/14/2022, 6:59 PM
No worries at all!
I would say that a lot of Prefect users are not even doing data pipelines. I have seen it as a way to monitor report generation or even CI/CD (when an event is triggered, run this Flow). Prefect by nature is super general purpose and as long as you can do it in Python, you can do it in Prefect for the most part. Dagster is very into data and lineage and, in general, you might have to conform more to get the most out of it. Going feature by feature is not a good approach in the workflow orchestration space because comparing Prefect and Airflow, the feature list has a lot of parity of course. People just want to run things on Docker or run things on Kubernetes. They want things to run in parallel at scale (Celery or Airflow, Dask for Prefect). But the main driver I have seen for people migrating from Airflow to Prefect has been ease of adoption. So a lot of it is about implementation over just the feature list.
There are some users here that use it as an interface to trigger parameterized reporting. A non-technical user goes into the UI and selects parameters and then triggers a Flow Run
I would say that Prefect is commonly seen as more general-purpose than Dagster whereas Dagster is specifically a data orchestrator, but that doesn’t exclude us either from Data Science or ML Applications. In general, as long as you can do it in Python, you can bring it to Prefect
4 Views