https://prefect.io logo
#prefect-community
Title
# prefect-community
i

ibrahem

06/15/2022, 8:42 PM
Hello, Have any one managed to run Shell Task via SSH ? or submitted a Spark job to a Yarn cluster ?
1
k

Kevin Kho

06/15/2022, 8:43 PM
I have not seen using SSH, but I think I have seen people directly submit with
spark-submit
i

ibrahem

06/15/2022, 8:45 PM
Prefect is installed on a different server than Spark so I need a way to Submit job remotely.
k

Kevin Kho

06/15/2022, 8:47 PM
How would you do it without Prefect? Just trough ssh?
i

ibrahem

06/15/2022, 8:48 PM
Yes I would SSH to a gateway server and just submit-spark a python script that is in the gateway server.
k

Kevin Kho

06/15/2022, 8:50 PM
Can’t you submit remotely with something like this ?
a

Anna Geller

06/15/2022, 8:51 PM
there is an API you can hit, we had a convo about it with one user before
i

ibrahem

06/15/2022, 8:54 PM
Thanks Kevin for the article I will definitely check this approach if feasible. @Anna Geller I like the idea of using an API can u please share where I can find the API details
i

ibrahem

06/17/2022, 3:06 PM
Thanks Anna
3 Views