Hi Prefect Community! I think I saw in one of my deep dives an example of a ResourceManager that manages an EMR cluster to submit spark jobs. I have been searching for a good 2 hours now and can always only find managing Dask clusters via a resource manager. Does anyone have a pointer? Very much appreciated. 🙂

Hi @Luis Muniz I don’t have much knowledge in EMR but from a prefect resource manager perspective I could image in would look something like this:

@resource_manager(name="spark")
class SparkCluster:
    def __init__(self, ...):
        self.... = ...

    def setup(self):
        return SparkClient(self...)

    def cleanup(self, client):
        client.close()

this the video you are thinking of?

YES!