Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Anyone integrating Prefect 3 with The <https://www.kubeflow.org/docs/components/spark-operator/overview/|Kubeflow Kubernetes Operator for Apache Spark>?

I'm new to Prefect (about a week) and Spark (a while longer) and have only a passing knowledge of kubernetes, so please bear with me if my question is basic:

Assuming my spark-operator is configured to run Spark jobs with Prefect in mind, and since the operator will persist in my Kubernetes cluster and itself create the pods it needs to execute its jobs, then do I need a Kubernetes work pool in Prefect at all? Or should I just use a Process work pool with local storage to find my flows on the Operator?

I find the lack of response here interesting. Not sure if I should interpret it as a negative response to my question whether anyone is integrating Prefect with the Kubeflow spark-operator.

I clearly have much to learn here, and learn I will. My question now is whether this is the place to put my questions and learnings, or take it offline.