How does Prefect compare to Kubeflow

Question

Chris White · Answer

Kubeflow is very much targeted at ML workflows on Kubernetes and is great for that use case the two biggest differences that you ll probably notice between the two is that Kubeflow will always greedily write data to disk in between tasks which can be either a feature or an annoyance depending on your workflow Kubeflow doesn t natively ship with a scheduler by scheduler here I mean something which allows you to specify future non ad hoc run times in an automated way

Chris White · Answer

Prefect was designed to be as workflow agnostic as possible and our goal is to support an incredibly diverse set of use cases ranging from your standard regularly scheduled Batch ETL jobs to ad hoc ML pipelines to scheduled reporting we also designed Prefect to be as lightweight as possible out of the box requiring nothing more than a python environment but with all the necessary hooks and features to scale up to a beefier deployment on systems like Kubernetes

Jason · Answer

i see thanks for the explanation < Chris White> I think one of the benefits of kube is that its all docker task based We have certain logic implemented in other languages so its nice to be able to leverage standard containers vs python only

Jason · Answer

however i saw that there is a docker task available

Jason · Answer

so that s good

Chris White · Answer

yea that s a good point each task in KubeFlow can have a very different environment while we are working on per task docker environments for Prefect it is not available right now

Chris White · Answer

we usually recommend people use the docker python SDK or the kubernetes python SDK for executing arbitrary docker based jobs