Has anyone used Prefect to spin up an EMR cluster ...
# ask-community
s
Has anyone used Prefect to spin up an EMR cluster with spark to do some processing, and then shut it all down? Seems similar to the ECS worker pool, but obviously with EMR instead.
o
Yes. I used
awswrangler
for talking with the cluster. Also had to create a Prefect task for waiting for the steps in cluster to be completed before shutting it down.
s
Ah, using wrangler is smart, good shout!
👍 1