Does anyone currently hook Prefect up to a spark cluster? We're using delta lake for data pipelines which uses spark under the hood and we're currently just running jobs on nodes with lots of memory, but quickly approaching the point where we'll need a spark cluster.
Does anyone have any experience / recommendations to share here?