Hi Guys, wondering if anyone did work with Spark on Prefect. Is there a way to pass spark dataframes between two tasks? I am trying to run Spark in client mode and pass the dataframes between tasks.

I don’t believe you can pickle a spark dataframe iirc. Prefect uses cloud pickle to pass data between tasks.

Hey @Omar Sultan, you would need to persist it somewhere (like as a parquet) and then load it in downstream tasks

you probably don’t want to do that tbh, you usually don’t want to pull data back to the driver which would be required to serialize it in order to pass between tasks

I think when using Spark in client mode, you’re sending instructions to the remote cluster and saving as parquet is performed by the partitions so it won’t force a collect. And on the Prefect side, as long as you return the location of the file as opposed to the data, it’s just the location string being serialized.

Hi guys thanks a lot for your replies

That’s what we do!

Thanks a lot Zach