Hi guys, I'm very curious about prefect.
We are build our big data architecture, no we use aws glue with pyspark, but we want to use aws emr, so we looking for a orchestration tool. Prefect can submit spark jobs to emr?
04/30/2020, 12:47 AM
Hi @Rubens Sôto! Anything you can do with Python, you can do with Prefect, including interfacing with EMR Spark! 😄 So yes, you can certainly use Prefect Tasks to orchestrate sending Spark jobs to EMR!
04/30/2020, 12:48 AM
But there is no native connector like airflow does?
04/30/2020, 12:51 AM
No we don't have a native connector but that sounds like a great addition to the Task Library!