Hey.. I am registering my task on prefect-server to demo project. The task has spark and sparks context. The flow.run() is successful but flow.register() gives following error
RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
Any guide on this spark context setting.