Hi folks, absolute newbie here so be gentle. I am looking to onboard prefect on to our current solution which uses databricks on Azure. I am aware of the prefect databricks methods for submitting jobs etc, and am reasonably comfortable with how this can work in our process with prefect. Where I am unsure is the local development story. For example, if I use the methods provided within prefect I guess I will have to submit a job to a databricks cluster. Without doing any sort or ‘if this then that’ condition, would it be possible for me to trigger a flow using prefect and submit the jobs to a local spark cluster, or do the nice methods provided by prefect only allow me to submit my job to a remote cluster? Thanks

Hi @Michael Law! How would you submit t o the local cluster without Prefect?

Thanks for getting back to me @Kevin Kho. As I understand it using the prefect api I can call the databricksrunsubmit method, but this couldn’t be used locally I would guess as it’s looking for a databricks cluster. So that’s why I’m scratching my head. As you say, databricks local development is hard

I think the answer here is to develop locally without access to the cluster, just on local Spark. Host the Prefect agent in a different machine that is configured to use `databricks-connect`so that when it runs Spark code, it hits the databricks cluster