Hello, Have any one managed to run Shell Task via SSH ? or submitted a Spark job to a Yarn cluster ?
✅ 1
k
Kevin Kho
06/15/2022, 8:43 PM
I have not seen using SSH, but I think I have seen people directly submit with
spark-submit
i
ibrahem
06/15/2022, 8:45 PM
Prefect is installed on a different server than Spark so I need a way to Submit job remotely.
k
Kevin Kho
06/15/2022, 8:47 PM
How would you do it without Prefect? Just trough ssh?
i
ibrahem
06/15/2022, 8:48 PM
Yes I would SSH to a gateway server and just submit-spark a python script that is in the gateway server.
k
Kevin Kho
06/15/2022, 8:50 PM
Can’t you submit remotely with something like this ?
a
Anna Geller
06/15/2022, 8:51 PM
there is an API you can hit, we had a convo about it with one user before
i
ibrahem
06/15/2022, 8:54 PM
Thanks Kevin for the article I will definitely check this approach if feasible. @Anna Geller I like the idea of using an API can u please share where I can find the API details