https://prefect.io logo
Title
j

Jason

10/05/2019, 3:09 PM
How does Prefect compare to Kubeflow?
c

Chris White

10/05/2019, 4:53 PM
Kubeflow is very much targeted at ML workflows on Kubernetes, and is great for that use case; the two biggest differences that you’ll probably notice between the two is that: - Kubeflow will always greedily write data to disk in between tasks (which can be either a feature or an annoyance depending on your workflow) - Kubeflow doesn’t natively ship with a scheduler (by “scheduler” here I mean something which allows you to specify future, non ad-hoc run times in an automated way)
Prefect was designed to be as workflow-agnostic as possible, and our goal is to support an incredibly diverse set of use cases, ranging from your standard regularly scheduled Batch ETL jobs, to ad-hoc ML pipelines, to scheduled reporting; we also designed Prefect to be as lightweight as possible out-of-the-box (requiring nothing more than a python environment), but with all the necessary hooks and features to scale up to a beefier deployment on systems like Kubernetes
j

Jason

10/05/2019, 5:39 PM
i see. thanks for the explanation @Chris White I think one of the benefits of kube is that its all docker task based. We have certain logic implemented in other languages so its nice to be able to leverage standard containers vs python only
however, i saw that there is a docker task available
so that's good
c

Chris White

10/05/2019, 5:40 PM
yea, that’s a good point - each task in KubeFlow can have a very different environment; while we are working on per-task docker environments for Prefect, it is not available right now
we usually recommend people use the docker python SDK or the kubernetes python SDK for executing arbitrary docker-based jobs