https://prefect.io logo
#prefect-community
Title
# prefect-community
c

Chris Hart

07/26/2019, 6:46 PM
this seems to be the most analogous example for pagination looping with tasks: https://docs.prefect.io/guide/tutorials/advanced-mapping.html#scaling-to-all-episodes only instead of URL lists, I'm mapping cursors and offsets
c

Chris White

07/26/2019, 8:59 PM
Yea this would be my vote --> to map over offsets of some kind. However, there is another option; I’m nervous to recommend it for this particular use case but I’d be interested to know if it suits your needs: you could do page 1 processing, and then raise a
RETRY
signal (https://docs.prefect.io/api/unreleased/engine/signals.html#retry) with a custom message / result. To determine which RETRY your task is currently on, you can pull
task_run_count
out of
prefect.context
full caveat: this is not the sort of application the signal is intended for, but it just might work for you
although…. I’m not sure how you would aggregate the results of all the individual task runs
c

Chris Hart

07/27/2019, 3:20 PM
🤔 thanks for the reply, will have a look at that approach!
very interesting.. one extension question to this is, I'm using the "functional api".. would switching to "imperative" allow me to access the values in a return value from a task? (thinking if so, could just manually loop the actual tasks)
c

Chris White

07/27/2019, 6:21 PM
Yea let me know! And no, switching APIs wouldn’t change anything - the underlying implementation / DAG details are identical between the two APIs, choosing one or the other is purely for personal preference
j

Jeremiah

07/27/2019, 7:19 PM
@Chris Hart — just had a great convo with @Chris White that resulted in this proposal: https://github.com/PrefectHQ/prefect/issues/1280 for first-class looping of single tasks. This is a surprisingly simple proposal — we also have some thoughts on extending this to a general (multi-task) case, but it’s more complex
c

Chris Hart

07/27/2019, 7:36 PM
awesome! thanks so much for looking into it
I was going to try out manual looping with the "imperative api" (instead of the context manager style I'm currently using)... but can you confirm that that the outer Flow object would not contain the task return values anywhere?
according to this proposal description it sounds like the only option is to do the looping inside the task
j

Jeremiah

07/27/2019, 7:46 PM
I suspect that this proposal would be the right framework for your use case, as you would be able to use the loop to discover and return results -- I'm not entirely sure what your objectives withe the imperative API is, though, as it has all the same control flow structures as the functional one. Let's see if we get this proposal implemented in the near term and work to meet your use case
🎉 1