Hello! I am trying to run a simple spark-operator job on prefect server by following the instructions provided here:
https://github.com/tardunge/prefect-spark-on-k8s-operator/blob/main/README.md. Attaching the flow and spark-operator yaml to this message. When I try to run the job locally (python3.11 job.py), I run into "prefect.exceptions.ScriptError: Script at 'simple-so-k8s-on-prefect.py' encountered an exception: TypeError("Object of type 'FieldInfo' is not JSON serializable")". Attaching entire stack trace for reference. Any idea what's happening? Note: I am running the code inside a container.
Slack Conversation