Hi Folks, I was wondering if anyone has used prefect to schedule spark jobs via EMR. This is the flow I am working on: but i want a way for step 1 ( create EMR cluster) to start when i rerun failed task ( step 2). does anyone have ideas other than involving step 1 from step 2. Thanks in advance .

Did not try them myself but I think ResourceManager is the way to go: https://docs.prefect.io/api/latest/tasks/resources.html#resourcemanager

Thanks @Michael Ludwig, ill have a look 👍