https://prefect.io logo
Title
r

Rubens Sôto

04/30/2020, 12:39 AM
Hi guys, I'm very curious about prefect. We are build our big data architecture, no we use aws glue with pyspark, but we want to use aws emr, so we looking for a orchestration tool. Prefect can submit spark jobs to emr? Thank you
n

nicholas

04/30/2020, 12:47 AM
Hi @Rubens Sôto! Anything you can do with Python, you can do with Prefect, including interfacing with EMR Spark! 😄 So yes, you can certainly use Prefect Tasks to orchestrate sending Spark jobs to EMR!
r

Rubens Sôto

04/30/2020, 12:48 AM
But there is no native connector like airflow does?
n

nicholas

04/30/2020, 12:51 AM
No we don't have a native connector but that sounds like a great addition to the Task Library!