Just a note to say, props to @Kevin Kho for giving me some pro-tips on Prefect setup in combination with databricks using delta lake. I have orchestrated a bunch of our jobs through our data layers now and setup a strategy for our developer loop using a combination of a local cluster for unit tests and databricks-connect for our on cluster runs, either via local machines or deployed to AKS. Not production ready yet, but huge strides. Couldnt have done it without your help ❤️
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
