What's the best way to incrementally load tables (...
# ask-community
j
What's the best way to incrementally load tables (e.x. a loop) but then also ensure that if a connection is lost, the task retry will re-establish the connection AND not create duplicates on previously loaded rows?
k
You can persist a watermark in the KV store to keep track of the last processed date then update it after an operation
j
Thank you!