https://prefect.io logo
n

Nakul Gowdra

10/08/2020, 1:58 AM
Hi Folks, I was wondering if anyone has used prefect to schedule spark jobs via EMR. This is the flow I am working on: but i want a way for step 1 ( create EMR cluster) to start when i rerun failed task ( step 2). does anyone have ideas other than involving step 1 from step 2. Thanks in advance .
m

Michael Ludwig

10/08/2020, 4:58 AM
Did not try them myself but I think ResourceManager is the way to go: https://docs.prefect.io/api/latest/tasks/resources.html#resourcemanager
n

Nakul Gowdra

10/08/2020, 10:25 AM
Thanks @Michael Ludwig, ill have a look 👍