Hello, Have any one managed to run Shell Task via SSH ? or submitted a Spark job to a Yarn cluster ?

I have not seen using SSH, but I think I have seen people directly submit with

spark-submit

Prefect is installed on a different server than Spark so I need a way to Submit job remotely.

How would you do it without Prefect? Just trough ssh?

Yes I would SSH to a gateway server and just submit-spark a python script that is in the gateway server.

Can’t you submit remotely with something like this ?

there is an API you can hit, we had a convo about it with one user before

Thanks Kevin for the article I will definitely check this approach if feasible. @Anna Geller I like the idea of using an API can u please share where I can find the API details

check this https://sparkbyexamples.com/spark/submit-spark-job-via-rest-api/

Thanks Anna