I'd be very curious to discuss a few points/user experience with a couple of people if possible/before committing.
j
Joe Schmid
10/29/2019, 7:35 PM
Hi @Arlo Bryer, we've been using Prefect Core (~4 months) and Cloud (~2 months) and I'd be happy to answer any questions and/or share our experience. (Our previous experience with data pipelines was using Airflow for about 4 years: we have hundreds of Airflow DAGs, etc.) Let me know what would be most useful for you to know.
a
Arlo Bryer
10/30/2019, 9:50 AM
Hi @Joe Schmid - great, thanks!
Arlo Bryer
10/30/2019, 9:50 AM
Have you completely moved away from Airflow in favour of Prefect now?
j
Joe Schmid
10/30/2019, 2:51 PM
@Arlo Bryer we are in the process of adopting Prefect, but for us it'll be a long process just since we have so many Airflow DAGs that we've built over 4 years. Right now we're using Prefect mainly for data science pipelines, while the existing data engineering pipelines remain on Airflow. Long-term though I don't see any reason why we couldn't use Prefect for all of our data pipelines, both for data engineering and data science.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.