Does anyone currently hook Prefect up to a spark cluster? We're using delta lake for data pipelines which uses spark under the hood and we're currently just running jobs on nodes with lots of memory, but quickly approaching the point where we'll need a spark cluster.
Does anyone have any experience / recommendations to share here?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
