hello everybody, I'm using Prefect to ETL data. But I don't know the strategy to follow yet:
1/ How many records were extracted and loaded?
2/ How to detect data need to baseline(new or update from source table) ?
1 year ago
Hey @Anh Nguyen, Prefect scales how to whatever your hardware can handle.For the detect data for a new baseline, you can store watermarks in the KV Store to load and continue the job from where it left off