Hey all I m a data engineer and functional programmer Lookin

Question

Hey all I m a data engineer and functional programmer Looking into Prefect as a viable alternative to Airflow after coming across this blog post <https medium com the prefect blog why not airflow 4cfa423299c4> The list of things Airflow doesn t support natively hit home with my own past experiences

Kevin Kho · Answer

Hi < Josiah Berkebile> Welcome to the community

Dylan · Answer

Welcome Josiah

Josiah Berkebile · Answer

So just looked through the docs at the scheduling and I LOVE this <https docs prefect io api latest schedules clocks html datesclock> My current place of work we have a lot of jobs that only need to run on specific calendar dates and we ve had to work around Airflow s cron style only scheduling to accomplish that we basically had to create a task upfront that checks the date and shortcuts the DAG if it s not any of the dates we want to run on It s fine but you can t see that scheduling rule from the Airflow UI which is kinda a pain

Dylan · Answer

Glad to hear it smile

Josiah Berkebile · Answer

Do you guys have a docker container or better a Helm chart that I could use to get things up and running for trialing real quick

Josiah Berkebile · Answer

I feel like I need to start tinkering with this so I can start talking to some of the companies that use Airflow in my city about maybe switching to Prefect

Josiah Berkebile · Answer

which brings up another question I noticed Prefect isn t version 1 0 yet Does that mean it s not really ready yet for production environments in a large enterprise

Kevin Kho · Answer

Here are the <https hub docker com r prefecthq prefect|docker images> We have large enterprise users

Josiah Berkebile · Answer

If I build a Helm Chart I ll pass it along to you guys What license would you want on something like that

Josiah Berkebile · Answer

I m inclined towards the AGPL but I would concede to MIT or Apache or something if you wanted

Kevin Kho · Answer

I think you can use the <https github com dask helm chart blob main dask values yaml|Dask helm chart> and just replace with the appropriate Prefect image

Josiah Berkebile · Answer

Thanks I ll start there when I get to the k8s implementation side of things once I m done toying with single node instances

Josiah Berkebile · Answer

is Prefect built on Dask

Kevin Kho · Answer

Yes the base prefect image will contain Dask

Josiah Berkebile · Answer

wow I suppose that could make it a bit more convenient to do certain types of data science workloads natively in Prefect since Dask would already be included as a library in the ecosystem

Kevin Kho · Answer

Yes Prefect integrates natively with Dask and has a `DaskExecutor` to parallelize tasks

Josiah Berkebile · Answer

this is definitely worth diving into I m really happy you guys decided to tackle this project

Josiah Berkebile · Answer

I ve been thinking ever since I started using Airflow that someone needs to build a better Airflow

Josiah Berkebile · Answer

the only thing I kinda miss for this sort of thing is type checked error handling since it s kinda hard to run and test workflow code that is highly integrated with hard to mock services that are running in the deployment environments

Josiah Berkebile · Answer

but that s such a nit pick and I m not convinced enough people in the industry value type safety enough to adopt a system like that anyway

Marko Herkaliuk · Answer

and here server alternative for cloud on your infra helm charts <https github com PrefectHQ server tree master helm prefect server>

Zach Schumacher · Answer

we are in the process of moving from Airflow to Prefect over at Simplebet FWIW

Matthew Alhonte · Answer

Yeah the native integration of Prefect + Dask vs the weirder marriage of Airflow + Spark is one of the big selling points imho