Hi Prefect experts, I want to try out Prefect in our existing ETL pipeline for scheduling/Spark job management - I know Prefect is a best match for Python based scripts ( eg: Pyspark) , would it support Spark Scala/Java jobs as well ? because our ETL is mainly built with Scala Spark jobs? any examples or document related to this matter? Thank you in advance! 🙏 ( sorry, if this is a duplicate question)

Hi Ajith, Prefect can run jobs of any type. For launching things other than Python tasks, you'd need some way to kick off a spark job from Python or using a shell script (and our

ShellTask

). This might call

spark-submit

or something else.

Thank you @Jim Crist-Harif for the quick response! If I got you correct - I need some kind of wrapper which can trigger Scala spark jobs based on Prefect calls? Is my understanding correct?

Yes, that's correct.

alright! thank you very much!