Hello! I am trying to run a simple spark-operator job on prefect server by following the instructions provided here: https://github.com/tardunge/prefect-spark-on-k8s-operator/blob/main/README.md. Attaching the flow and spark-operator yaml to this message. When I try to run the job locally (python3.11 job.py), I run into "prefect.exceptions.ScriptError: Script at 'simple-so-k8s-on-prefect.py' encountered an exception: TypeError("Object of type 'FieldInfo' is not JSON serializable")". Attaching entire stack trace for reference. Any idea what's happening? Note: I am running the code inside a container. Slack Conversation

haven’t looked deeply but i recognize that error as “someone put a v2 pydantic field on a v1 model” error

I see that someone has raised this issue already: https://github.com/tardunge/prefect-spark-on-k8s-operator/issues/8