https://prefect.io logo
Title
j

Josiah Berkebile

05/12/2021, 3:46 PM
Hey all, I'm a data engineer and functional programmer. Looking into Prefect as a viable alternative to Airflow after coming across this blog post: https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 The list of things Airflow doesn't support natively hit home with my own past experiences
👋 8
k

Kevin Kho

05/12/2021, 4:01 PM
Hi @Josiah Berkebile! Welcome to the community!
❤️ 1
d

Dylan

05/12/2021, 4:37 PM
Welcome, Josiah!
❤️ 1
j

Josiah Berkebile

05/12/2021, 5:07 PM
So, just looked through the docs at the scheduling, and I LOVE this: https://docs.prefect.io/api/latest/schedules/clocks.html#datesclock My current place of work, we have a lot of jobs that only need to run on specific calendar dates, and we've had to work around Airflow's cron-style-only scheduling to accomplish that (we basically had to create a task upfront that checks the date and shortcuts the DAG if it's not any of the dates we want to run on. It's fine, but you can't see that scheduling rule from the Airflow UI, which is kinda a pain).
❤️ 1
d

Dylan

05/12/2021, 5:35 PM
Glad to hear it! 😄
j

Josiah Berkebile

05/12/2021, 6:09 PM
Do you guys have a docker container, or better, a Helm chart that I could use to get things up and running for trialing real quick?
I feel like I need to start tinkering with this so I can start talking to some of the companies that use Airflow in my city about maybe switching to Prefect
which brings up another question, I noticed Prefect isn't version 1.0, yet. Does that mean it's not really ready yet for production environments in a large enterprise?
k

Kevin Kho

05/12/2021, 6:10 PM
Here are the docker images . We have large enterprise users.
🙌 1
j

Josiah Berkebile

05/12/2021, 6:11 PM
If I build a Helm Chart, I'll pass it along to you guys. What license would you want on something like that?
I'm inclined towards the AGPL, but I would concede to MIT or Apache or something if you wanted
k

Kevin Kho

05/12/2021, 6:12 PM
I think you can use the Dask helm chart and just replace with the appropriate Prefect image.
👍 1
j

Josiah Berkebile

05/12/2021, 6:12 PM
Thanks, I'll start there when I get to the k8s implementation side of things once I'm done toying with single-node instances
is Prefect built on Dask?
k

Kevin Kho

05/12/2021, 6:13 PM
Yes the base prefect image will contain Dask
j

Josiah Berkebile

05/12/2021, 6:16 PM
wow. I suppose that could make it a bit more convenient to do certain types of data science workloads natively in Prefect since Dask would already be included as a library in the ecosystem
👍 2
k

Kevin Kho

05/12/2021, 6:17 PM
Yes Prefect integrates natively with Dask and has a
DaskExecutor
to parallelize tasks.
j

Josiah Berkebile

05/12/2021, 6:18 PM
this is definitely worth diving into. I'm really happy you guys decided to tackle this project.
:thank-you: 1
I've been thinking ever since I started using Airflow that someone needs to build a better Airflow
the only thing I kinda miss for this sort of thing is type-checked error handling since it's kinda hard to run and test workflow code that is highly integrated with hard-to-mock services that are running in the deployment environments
but, that's such a nit-pick, and I'm not convinced enough people in the industry value type-safety enough to adopt a system like that anyway
m

Marko Herkaliuk

05/12/2021, 7:22 PM
and here server (alternative for cloud on your infra) helm charts - https://github.com/PrefectHQ/server/tree/master/helm/prefect-server
👍 1
🙌 1
z

Zach Schumacher

05/14/2021, 5:28 PM
we are in the process of moving from Airflow to Prefect over at Simplebet, FWIW
👍 1
m

Matthew Alhonte

05/14/2021, 9:23 PM
Yeah, the native integration of Prefect + Dask vs the weirder marriage of Airflow + Spark is one of the big selling points imho.