Still coming to Databricks and I guess in general with Tasks. Databricks has added git support for jobs, and the current DatabricksSubmitMultitaskRun doesn't support it. I am in doubt among the possible approaches:
• Create a DatabricksSubmitMultitaskRun custom implementation
• Use the databricks CLI python library to create a job and then just run it via Prefect
• Other?
The real problem is that the Task does not allow dependency injection (i.e. the databricks client is created within the Run function, so it's not easy to override it). I guess the design of the Task is concerning in the sense that is not extensible, one needs to rewrite it from scratch
k
Kevin Kho
05/05/2022, 1:20 AM
Can you post this in #CL09KU1K7? This channel is not watched by the integrations engineers because it’s about best practices. And then I can point them to the message
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.