Dominic Pham

    Dominic Pham

    10 months ago
    Hi all, given that during flow_run, task_A will return a list from a sql query downstream to task_B. task_B will then start at index[0] and send that value downstream to task_C. Now, x minutes later, flow_run will run again, and task_A will return the same list, but task_B will now return index[1] instead of index[0] and send the value of index[1] to task_C and so on and so forth. What would be the best approach for this, where I want a given flow, between scheduled runs, to remember its location in an iterable?
    Michael Adkins

    Michael Adkins

    10 months ago
    Hey @Dominic Pham — this sounds like a great use for the KV store https://docs.prefect.io/orchestration/concepts/kv_store.html
    Just pull the index for each run, increment it, and push the new value
    Dominic Pham

    Dominic Pham

    10 months ago
    Thanks for the response, I'll take a look into this!
    Would there be a different solution for Prefect Core? This solution for cloud looks like it could work fine, but I also want to explore a solution for Core as well if possible
    Kevin Kho

    Kevin Kho

    10 months ago
    I think you just need to persist the counter in a file and then read and write from it
    Michael Adkins

    Michael Adkins

    10 months ago
    You could also update a parameter default for the flow at the end of each flow run, but that's a hack