Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all, given that during flow_run, task_A will return a list from a sql query downstream to task_B. task_B will then start at index[0] and send that value downstream to task_C.
Now, x minutes later, flow_run will run again, and task_A will return the same list, but task_B will now return index[1] instead of index[0] and send the value of index[1] to task_C and so on and so forth.
What would be the best approach for this, where I want a given flow, between scheduled runs, to remember its location in an iterable?

Hey <@U02E4DNHPMM> — this sounds like a great use for the KV store <https://docs.prefect.io/orchestration/concepts/kv_store.html>

Just pull the index for each run, increment it, and push the new value

Thanks for the response, I'll take a look into this!

Would there be a different solution for Prefect Core? This solution for cloud looks like it could work fine, but I also want to explore a solution for Core as well if possible

I think you just need to persist the counter in a file and then read and write from it

You could also update a parameter default for the flow at the end of each flow run, but that's a hack