this seems to be the most analogous example for pa...
# prefect-communityc
Chris Hart
07/26/2019, 6:46 PMthis seems to be the most analogous example for pagination looping with tasks: https://docs.prefect.io/guide/tutorials/advanced-mapping.html#scaling-to-all-episodes only instead of URL lists, I'm mapping cursors and offsets
c
Chris White
07/26/2019, 8:59 PM
Yea this would be my vote --> to map over offsets of some kind. However, there is another option; I’m nervous to recommend it for this particular use case but I’d be interested to know if it suits your needs: you could do page 1 processing, and then raise a
RETRYtask_run_countprefect.contextChris White
07/26/2019, 8:59 PM
full caveat: this is not the sort of application the signal is intended for, but it just might work for you
Chris White
07/26/2019, 9:00 PM
although…. I’m not sure how you would aggregate the results of all the individual task runs
c
Chris Hart
07/27/2019, 3:20 PM🤔 thanks for the reply, will have a look at that approach!
Chris Hart
07/27/2019, 3:29 PMvery interesting.. one extension question to this is, I'm using the "functional api".. would switching to "imperative" allow me to access the values in a return value from a task? (thinking if so, could just manually loop the actual tasks)
c
Chris White
07/27/2019, 6:21 PM
Yea let me know! And no, switching APIs wouldn’t change anything - the underlying implementation / DAG details are identical between the two APIs, choosing one or the other is purely for personal preference
j
Jeremiah
07/27/2019, 7:19 PM
@Chris Hart — just had a great convo with @Chris White that resulted in this proposal: https://github.com/PrefectHQ/prefect/issues/1280 for first-class looping of single tasks. This is a surprisingly simple proposal — we also have some thoughts on extending this to a general (multi-task) case, but it’s more complex
c
Chris Hart
07/27/2019, 7:36 PMawesome! thanks so much for looking into it
Chris Hart
07/27/2019, 7:38 PMI was going to try out manual looping with the "imperative api" (instead of the context manager style I'm currently using)... but can you confirm that that the outer Flow object would not contain the task return values anywhere?
Chris Hart
07/27/2019, 7:38 PMaccording to this proposal description it sounds like the only option is to do the looping inside the task
j
Jeremiah
07/27/2019, 7:46 PM
I suspect that this proposal would be the right framework for your use case, as you would be able to use the loop to discover and return results -- I'm not entirely sure what your objectives withe the imperative API is, though, as it has all the same control flow structures as the functional one. Let's see if we get this proposal implemented in the near term and work to meet your use case
🎉 1