Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Has anyone used Prefect to spin up an EMR cluster with spark to do some processing, and then shut it all down? Seems similar to the ECS worker pool, but obviously with EMR instead.

Yes. I used `awswrangler` for talking with the cluster. Also had to create a Prefect task for waiting for the steps in cluster to be completed before shutting it down.