Deepak
09/28/2022, 11:08 PMRuntimeError: Java gateway process exited before sending its port number
when I run pyspark methods inside a task.Matt Conger
09/28/2022, 11:22 PMDeepak
09/29/2022, 4:19 PMdf_spark = ps.from_pandas(df_pandas)
, which is where it is failing.I tried two things
First was to execute export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell"
before running prefect register.
Second was to add this snippet
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
df_pandas = sc.parallelize(df_pandas)
before df_spark = ps.from_pandas(df_pandas)
but neither of these worked. I got the same error, i.e. Java gateway process exited before sending its port number
.Matt Conger
09/30/2022, 10:13 PM