We also have problems with the logs in the UI. We use Prefect-Databricks which exposes a flow called “runs and submit”. This flow invoke a flow “wait result” with timeout in loop. On the UI, the job ID of the databricks job is not visible in the upper level job, and you need to click several times before you find it
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
