Hi Prefect experts, I want to try out *Prefect* in...
# prefect-community
a
Hi Prefect experts, I want to try out Prefect in our existing ETL pipeline for scheduling/Spark job management - I know Prefect is a best match for Python based scripts ( eg: Pyspark) , would it support Spark Scala/Java jobs as well ? because our ETL is mainly built with Scala Spark jobs? any examples or document related to this matter? Thank you in advance! 🙏 ( sorry, if this is a duplicate question)
j
Hi Ajith, Prefect can run jobs of any type. For launching things other than Python tasks, you'd need some way to kick off a spark job from Python or using a shell script (and our
ShellTask
). This might call
spark-submit
or something else.
a
Thank you @Jim Crist-Harif for the quick response! If I got you correct - I need some kind of wrapper which can trigger Scala spark jobs based on Prefect calls? Is my understanding correct?
j
Yes, that's correct.
a
alright! thank you very much!