Hey, we've recently started exploring Prefect and we'd like to schedule Spark jobs on Cloud Dataproc - I've noticed the excellent integration with Databricks but didn't see anything re. running jobs on other Spark platforms/distributions. Am I missing something? Has anyone implemented this already? Thanks a lot for your help!

Hi @Remi Paulin, no we don’t have other Tasks in our library for Spark. You can add a feature request on Github so other people can voice their support of this feature. Databricks is already available on GCP (though I understand if you don’t want to use it in favor of Dataproc)

Hi @Kevin Kho thanks a lot for the quick reply! Ok, got it, we might actually use Databricks on GCP, especially if integration with Prefect is more straightforward. Thanks again!

We’d gladly accept the Cloud Dataproc task integration if ever you guys do build one though!