https://prefect.io logo
Title
e

Erick Heredia

05/17/2021, 12:59 PM
Hi all. I'm Erick. I'm a Data Scientist trying to wrap my head around Prefect.
๐Ÿ‘‹ 3
๐Ÿ™Œ 1
z

Zach Angell

05/17/2021, 1:01 PM
Welcome @Erick Heredia ๐Ÿ‘‹ !
e

Erick Heredia

05/17/2021, 1:05 PM
Thanks @Zach Angell. I have a question regarding this platform. Our team is also benchmarking ClearML for our operations. Do you know that tech and if so, what would be the differences between Prefect and ClearML?
z

Zach Angell

05/17/2021, 1:10 PM
I don't know ClearML well enough to give a good answer here. I'll check with the rest of the team to see if anyone is familiar. If you toss your question in the #prefect-community channel, you'll get good answers from both our team and Prefect community members who may have compared the two offerings!
e

Erick Heredia

05/17/2021, 1:20 PM
Will do, thanks.
k

Kevin Kho

05/17/2021, 1:21 PM
Hi @Erick Heredia, welcome to Prefect! Which ClearML product are you looking into? I assume Experiment and Orchestrate?
e

Erick Heredia

05/17/2021, 1:24 PM
Yes @Kevin Kho.
We're planning on using Feast for Feature Store, BentoML for deployment and ClearML for Experiments and Orchestration. We want to know what would be the differences between ClearML and Prefect.
k

Kevin Kho

05/17/2021, 1:31 PM
So for experiment tracking, Prefect is more general purpose but we see users combining experiment tracking (like MLFlow) and Prefect. They register their experiments within their scheduled flows, assuming there is a programmatic API to do so. For orchestration, it looks like ClearML lets you queue up training and manage compute power. I think Prefect is more meant for jobs that run on a schedule while ClearML is meant more for that job queue. There are Prefect mechanisms to handle hardware. Both of them let you run code on-premise if thatโ€™s a concern.
In general, if you are tying together all of those services with Python code that needs to be run on a schedule (or even on-demand), then Prefect is used for that macro orchestration level.
e

Erick Heredia

05/17/2021, 1:33 PM
Nice.
Thanks.