How does Prefect compare to Kubeflow?
# prefect-communityj
Jason
10/05/2019, 3:09 PMHow does Prefect compare to Kubeflow?
c
Chris White
10/05/2019, 4:53 PM
Kubeflow is very much targeted at ML workflows on Kubernetes, and is great for that use case; the two biggest differences that you’ll probably notice between the two is that:
- Kubeflow will always greedily write data to disk in between tasks (which can be either a feature or an annoyance depending on your workflow)
- Kubeflow doesn’t natively ship with a scheduler (by “scheduler” here I mean something which allows you to specify future, non ad-hoc run times in an automated way)
Chris White
10/05/2019, 4:55 PM
Prefect was designed to be as workflow-agnostic as possible, and our goal is to support an incredibly diverse set of use cases, ranging from your standard regularly scheduled Batch ETL jobs, to ad-hoc ML pipelines, to scheduled reporting; we also designed Prefect to be as lightweight as possible out-of-the-box (requiring nothing more than a python environment), but with all the necessary hooks and features to scale up to a beefier deployment on systems like Kubernetes
j
Jason
10/05/2019, 5:39 PMi see. thanks for the explanation @Chris White I think one of the benefits of kube is that its all docker task based. We have certain logic implemented in other languages so its nice to be able to leverage standard containers vs python only
Jason
10/05/2019, 5:39 PMhowever, i saw that there is a docker task available
Jason
10/05/2019, 5:39 PMso that's good
c
Chris White
10/05/2019, 5:40 PM
yea, that’s a good point - each task in KubeFlow can have a very different environment; while we are working on per-task docker environments for Prefect, it is not available right now
Chris White
10/05/2019, 5:40 PM
we usually recommend people use the docker python SDK or the kubernetes python SDK for executing arbitrary docker-based jobs