Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello, 

Trying to find any resources regarding Prefect and Spark (not on databricks but EMR)

Is there any example in docs showcasing how to submit spark job from prefect task ? 

Appreciate your help !!

Hey <@U01CWF05D6Y>, I think you can use `spark-submit` by passing it as a command through the Prefect ShellTask. what do you think of that? How would you normally submit? Using the command line tool?

Yes I am currently usung command line for spark-submit...ShellTask might work...I will check on how to pass environment variables and collect stdout and stderr within ShellTask...