Thread
#prefect-community
    ibrahem

    ibrahem

    3 months ago
    Hello, Have any one managed to run Shell Task via SSH ? or submitted a Spark job to a Yarn cluster ?
    Kevin Kho

    Kevin Kho

    3 months ago
    I have not seen using SSH, but I think I have seen people directly submit with
    spark-submit
    ibrahem

    ibrahem

    3 months ago
    Prefect is installed on a different server than Spark so I need a way to Submit job remotely.
    Kevin Kho

    Kevin Kho

    3 months ago
    How would you do it without Prefect? Just trough ssh?
    ibrahem

    ibrahem

    3 months ago
    Yes I would SSH to a gateway server and just submit-spark a python script that is in the gateway server.
    Kevin Kho

    Kevin Kho

    3 months ago
    Can’t you submit remotely with something like this ?
    Anna Geller

    Anna Geller

    3 months ago
    there is an API you can hit, we had a convo about it with one user before
    ibrahem

    ibrahem

    3 months ago
    Thanks Kevin for the article I will definitely check this approach if feasible. @Anna Geller I like the idea of using an API can u please share where I can find the API details
    Anna Geller

    Anna Geller

    3 months ago
    ibrahem

    ibrahem

    3 months ago
    Thanks Anna