Edmondo Porcu

    Edmondo Porcu

    4 months ago
    Still coming to Databricks and I guess in general with Tasks. Databricks has added git support for jobs, and the current DatabricksSubmitMultitaskRun doesn't support it. I am in doubt among the possible approaches: • Create a DatabricksSubmitMultitaskRun custom implementation • Use the databricks CLI python library to create a job and then just run it via Prefect • Other? The real problem is that the Task does not allow dependency injection (i.e. the databricks client is created within the Run function, so it's not easy to override it). I guess the design of the Task is concerning in the sense that is not extensible, one needs to rewrite it from scratch
    Kevin Kho

    Kevin Kho

    4 months ago
    Can you post this in #prefect-community? This channel is not watched by the integrations engineers because it’s about best practices. And then I can point them to the message
    Thank you!