this seems to be the most analogous example for pagination l

Question

this seems to be the most analogous example for pagination looping with tasks <https docs prefect io guide tutorials advanced mapping html scaling to all episodes> only instead of URL lists I m mapping cursors and offsets

Chris White · Answer

Yea this would be my vote gt to map over offsets of some kind However there is another option I m nervous to recommend it for this particular use case but I d be interested to know if it suits your needs you could do page 1 processing and then raise a `RETRY` signal <https docs prefect io api unreleased engine signals html retry> with a custom message result To determine which RETRY your task is currently on you can pull `task run count` out of `prefect context`

Chris White · Answer

full caveat this is not the sort of application the signal is intended for but it just might work for you

Chris White · Answer

although I m not sure how you would aggregate the results of all the individual task runs

Chris Hart · Answer

thinking face thanks for the reply will have a look at that approach

Chris Hart · Answer

very interesting one extension question to this is I m using the functional api would switching to imperative allow me to access the values in a return value from a task thinking if so could just manually loop the actual tasks

Chris White · Answer

Yea let me know And no switching APIs wouldn t change anything the underlying implementation DAG details are identical between the two APIs choosing one or the other is purely for personal preference

Jeremiah · Answer

< Chris Hart> just had a great convo with < Chris White> that resulted in this proposal for first class looping of single tasks This is a surprisingly simple proposal we also have some thoughts on extending this to a general multi task case but it s more complex

Chris Hart · Answer

awesome thanks so much for looking into it

Chris Hart · Answer

I was going to try out manual looping with the imperative api instead of the context manager style I m currently using but can you confirm that that the outer Flow object would not contain the task return values anywhere

Chris Hart · Answer

according to this proposal description it sounds like the only option is to do the looping inside the task

Jeremiah · Answer

I suspect that this proposal would be the right framework for your use case as you would be able to use the loop to discover and return results I m not entirely sure what your objectives withe the imperative API is though as it has all the same control flow structures as the functional one Let s see if we get this proposal implemented in the near term and work to meet your use case