We do have two problems with Prefect flow when using Prefect-Databricks:
• the job_sumit_and_wait flows uses subflows that ends up polluting the UI (who cares about how many times you polled to get the result from Databricks?)
• the information to find the Databricks job are not located in our parent job that really submit the job on Databricks, but we need to navigate to child flos
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.