Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi guys, I'm very curious about prefect.
We are build our big data architecture, no we use aws glue with pyspark, but we want to use aws emr, so we looking for a orchestration tool. Prefect can submit spark jobs to emr?

Thank you

Hi <@U012LBYDHHU>! Anything you can do with Python, you can do with Prefect, including interfacing with EMR Spark! :smile: So yes, you can certainly use Prefect Tasks to orchestrate sending Spark jobs to EMR!

But there is no native connector like airflow does?

No we don't have a native connector but that sounds like a great addition to the Task Library!