Josiah Berkebile

    Josiah Berkebile

    1 year ago
    Hey all, I'm a data engineer and functional programmer. Looking into Prefect as a viable alternative to Airflow after coming across this blog post: https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 The list of things Airflow doesn't support natively hit home with my own past experiences
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Josiah Berkebile! Welcome to the community!
    Dylan

    Dylan

    1 year ago
    Welcome, Josiah!
    Josiah Berkebile

    Josiah Berkebile

    1 year ago
    So, just looked through the docs at the scheduling, and I LOVE this:https://docs.prefect.io/api/latest/schedules/clocks.html#datesclock My current place of work, we have a lot of jobs that only need to run on specific calendar dates, and we've had to work around Airflow's cron-style-only scheduling to accomplish that (we basically had to create a task upfront that checks the date and shortcuts the DAG if it's not any of the dates we want to run on. It's fine, but you can't see that scheduling rule from the Airflow UI, which is kinda a pain).
    Dylan

    Dylan

    1 year ago
    Glad to hear it! 😄
    Josiah Berkebile

    Josiah Berkebile

    1 year ago
    Do you guys have a docker container, or better, a Helm chart that I could use to get things up and running for trialing real quick?
    I feel like I need to start tinkering with this so I can start talking to some of the companies that use Airflow in my city about maybe switching to Prefect
    which brings up another question, I noticed Prefect isn't version 1.0, yet. Does that mean it's not really ready yet for production environments in a large enterprise?
    Kevin Kho

    Kevin Kho

    1 year ago
    Here are the docker images . We have large enterprise users.
    Josiah Berkebile

    Josiah Berkebile

    1 year ago
    If I build a Helm Chart, I'll pass it along to you guys. What license would you want on something like that?
    I'm inclined towards the AGPL, but I would concede to MIT or Apache or something if you wanted
    Kevin Kho

    Kevin Kho

    1 year ago
    I think you can use the Dask helm chart and just replace with the appropriate Prefect image.
    Josiah Berkebile

    Josiah Berkebile

    1 year ago
    Thanks, I'll start there when I get to the k8s implementation side of things once I'm done toying with single-node instances
    is Prefect built on Dask?
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes the base prefect image will contain Dask
    Josiah Berkebile

    Josiah Berkebile

    1 year ago
    wow. I suppose that could make it a bit more convenient to do certain types of data science workloads natively in Prefect since Dask would already be included as a library in the ecosystem
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes Prefect integrates natively with Dask and has a
    DaskExecutor
    to parallelize tasks.
    Josiah Berkebile

    Josiah Berkebile

    1 year ago
    this is definitely worth diving into. I'm really happy you guys decided to tackle this project.
    I've been thinking ever since I started using Airflow that someone needs to build a better Airflow
    the only thing I kinda miss for this sort of thing is type-checked error handling since it's kinda hard to run and test workflow code that is highly integrated with hard-to-mock services that are running in the deployment environments
    but, that's such a nit-pick, and I'm not convinced enough people in the industry value type-safety enough to adopt a system like that anyway
    Marko Herkaliuk

    Marko Herkaliuk

    1 year ago
    and here server (alternative for cloud on your infra) helm charts - https://github.com/PrefectHQ/server/tree/master/helm/prefect-server
    Zach Schumacher

    Zach Schumacher

    1 year ago
    we are in the process of moving from Airflow to Prefect over at Simplebet, FWIW
    m

    Matthew Alhonte

    1 year ago
    Yeah, the native integration of Prefect + Dask vs the weirder marriage of Airflow + Spark is one of the big selling points imho.