Does anyone have tried run a pyspark ( spark clust...
# ask-communityq
Qin XIA
09/16/2021, 9:43 AMDoes anyone have tried run a pyspark ( spark cluster in local docker ) task from a local agent in Docker without using bash command ?
a
Andor T贸th
09/16/2021, 10:21 AMOnly in AirFlow
Andor T贸th
09/16/2021, 10:22 AMbut it should be no different in Prefect
Andor T贸th
09/16/2021, 10:25 AMThat's the gist of the Spark hook I'm using
Copy code
spark_conf = SparkConf()
spark_conf.setAll(self.conf.items())
spark_session = SparkSession.builder.config(conf=spark_conf).getOrCreate()Andor T贸th
09/16/2021, 10:27 AMthe prerequisite is to have a matching
pyspark馃檶 1
q
Qin XIA
09/16/2021, 11:24 AMi agree with you and that's what i want to try. But need to use spark-submit to run pyspark script, no (prefect flow would be run by python or python3 command)
a
Andor T贸th
09/16/2021, 11:30 AMso you want to submit a python script without using cli?
q
Qin XIA
09/16/2021, 11:30 AMI have tried add function findspark in my code and it works
a
Andor T贸th
09/16/2021, 11:51 AMCould you share the details!?
q
Qin XIA
09/16/2021, 12:52 PMI created a flow ( ex : flow.py) prefect who has some pyspark task. Creat all these elements as usual. Run this flow with command python flow.py, first time it donesn't work because PATH PYSPARK was not configured. So I just add function findspark in order to initialize PATH PYSPARK and it works.
馃憤 1
Qin XIA
09/16/2021, 12:53 PMI think if you configured your pyspark path in env variable, it should work also
馃檶 1
2 Views