Anh Nguyen

    Anh Nguyen

    1 year ago
    hello everybody, I'm using Prefect to ETL data. But I don't know the strategy to follow yet: 1/ How many records were extracted and loaded? 2/ How to detect data need to baseline(new or update from source table) ? Thanks all
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Anh Nguyen, Prefect scales how to whatever your hardware can handle. For the detect data for a new baseline, you can store watermarks in the KV Store to load and continue the job from where it left off
    Anh Nguyen

    Anh Nguyen

    1 year ago
    Thanks you! @Kevin Kho