Hi everybody, I am looking to implement a job outside of Databricks that can pull data from a remote API periodically, while keeping track of the next timestamp to use for each request, ie.

max(timestamp) + 1

in each returned response. Is prefect a good tool to use? I have been using databricks exclusively so this will be the first tool added to my "data stack", and it looks like I will need to move my scheduled job definitions out of databricks, and into prefect, with a step that invokes the old databricks task