Edmondo Porcu

05/05/2022, 1:18 AM
Still coming to Databricks and I guess in general with Tasks. Databricks has added git support for jobs, and the current DatabricksSubmitMultitaskRun doesn't support it. I am in doubt among the possible approaches: • Create a DatabricksSubmitMultitaskRun custom implementation • Use the databricks CLI python library to create a job and then just run it via Prefect • Other? The real problem is that the Task does not allow dependency injection (i.e. the databricks client is created within the Run function, so it's not easy to override it). I guess the design of the Task is concerning in the sense that is not extensible, one needs to rewrite it from scratch

Kevin Kho

05/05/2022, 1:20 AM
Can you post this in #prefect-community? This channel is not watched by the integrations engineers because it’s about best practices. And then I can point them to the message
upvote 1
Thank you!