Hey all, I'm a data engineer and functional progra...
# introductionsj
Josiah Berkebile
05/12/2021, 3:46 PMHey all, I'm a data engineer and functional programmer. Looking into Prefect as a viable alternative to Airflow after coming across this blog post:
https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4
The list of things Airflow doesn't support natively hit home with my own past experiences
👋 8
k
Kevin Kho
05/12/2021, 4:01 PM
Hi @Josiah Berkebile! Welcome to the community!
❤️ 1
d
Dylan
05/12/2021, 4:37 PM
Welcome, Josiah!
❤️ 1
j
Josiah Berkebile
05/12/2021, 5:07 PMSo, just looked through the docs at the scheduling, and I LOVE this:
https://docs.prefect.io/api/latest/schedules/clocks.html#datesclock
My current place of work, we have a lot of jobs that only need to run on specific calendar dates, and we've had to work around Airflow's cron-style-only scheduling to accomplish that (we basically had to create a task upfront that checks the date and shortcuts the DAG if it's not any of the dates we want to run on. It's fine, but you can't see that scheduling rule from the Airflow UI, which is kinda a pain).
❤️ 1
d
Dylan
05/12/2021, 5:35 PM
Glad to hear it! 😄
j
Josiah Berkebile
05/12/2021, 6:09 PMDo you guys have a docker container, or better, a Helm chart that I could use to get things up and running for trialing real quick?
Josiah Berkebile
05/12/2021, 6:09 PMI feel like I need to start tinkering with this so I can start talking to some of the companies that use Airflow in my city about maybe switching to Prefect
Josiah Berkebile
05/12/2021, 6:10 PMwhich brings up another question, I noticed Prefect isn't version 1.0, yet. Does that mean it's not really ready yet for production environments in a large enterprise?
k
Kevin Kho
05/12/2021, 6:10 PM
Here are the docker images . We have large enterprise users.
🙌 1
j
Josiah Berkebile
05/12/2021, 6:11 PMIf I build a Helm Chart, I'll pass it along to you guys. What license would you want on something like that?
Josiah Berkebile
05/12/2021, 6:12 PMI'm inclined towards the AGPL, but I would concede to MIT or Apache or something if you wanted
k
Kevin Kho
05/12/2021, 6:12 PM
I think you can use the Dask helm chart and just replace with the appropriate Prefect image.
👍 1
j
Josiah Berkebile
05/12/2021, 6:12 PMThanks, I'll start there when I get to the k8s implementation side of things once I'm done toying with single-node instances
Josiah Berkebile
05/12/2021, 6:12 PMis Prefect built on Dask?
k
Kevin Kho
05/12/2021, 6:13 PM
Yes the base prefect image will contain Dask
j
Josiah Berkebile
05/12/2021, 6:16 PMwow. I suppose that could make it a bit more convenient to do certain types of data science workloads natively in Prefect since Dask would already be included as a library in the ecosystem
👍 2
k
Kevin Kho
05/12/2021, 6:17 PM
Yes Prefect integrates natively with Dask and has a
DaskExecutorj
Josiah Berkebile
05/12/2021, 6:18 PMthis is definitely worth diving into. I'm really happy you guys decided to tackle this project.
🙏 1
Josiah Berkebile
05/12/2021, 6:19 PMI've been thinking ever since I started using Airflow that someone needs to build a better Airflow
Josiah Berkebile
05/12/2021, 6:21 PMthe only thing I kinda miss for this sort of thing is type-checked error handling since it's kinda hard to run and test workflow code that is highly integrated with hard-to-mock services that are running in the deployment environments
Josiah Berkebile
05/12/2021, 6:22 PMbut, that's such a nit-pick, and I'm not convinced enough people in the industry value type-safety enough to adopt a system like that anyway
m
Marko Herkaliuk
05/12/2021, 7:22 PMand here server (alternative for cloud on your infra) helm charts - https://github.com/PrefectHQ/server/tree/master/helm/prefect-server
👍 1
🙌 1
z
Zach Schumacher
05/14/2021, 5:28 PMwe are in the process of moving from Airflow to Prefect over at Simplebet, FWIW
👍 1
m
Matthew Alhonte
05/14/2021, 9:23 PMYeah, the native integration of Prefect + Dask vs the weirder marriage of Airflow + Spark is one of the big selling points imho.
4 Views