https://prefect.io
Join Slack
this seems to be the most analogous example for pagination looping with tasks: <https://docs.prefect...
# ask-community
c
this seems to be the most analogous example for pagination looping with tasks: https://docs.prefect.io/guide/tutorials/advanced-mapping.html#scaling-to-all-episodes only instead of URL lists, I'm mapping cursors and offsets
c
Yea this would be my vote --> to map over offsets of some kind. However, there is another option; I’m nervous to recommend it for this particular use case but I’d be interested to know if it suits your needs: you could do page 1 processing, and then raise a 
RETRY
signal (https://docs.prefect.io/api/unreleased/engine/signals.html#retry) with a custom message / result. To determine which RETRY your task is currently on, you can pull 
task_run_count
out of 
prefect.context
full caveat: this is not the sort of application the signal is intended for, but it just might work for you
although…. I’m not sure how you would aggregate the results of all the individual task runs
c
🤔 thanks for the reply, will have a look at that approach!
very interesting.. one extension question to this is, I'm using the "functional api".. would switching to "imperative" allow me to access the values in a return value from a task? (thinking if so, could just manually loop the actual tasks)
c
Yea let me know! And no, switching APIs wouldn’t change anything - the underlying implementation / DAG details are identical between the two APIs, choosing one or the other is purely for personal preference
j
@Chris Hart — just had a great convo with @Chris White that resulted in this proposal: https://github.com/PrefectHQ/prefect/issues/1280 for first-class looping of single tasks. This is a surprisingly simple proposal — we also have some thoughts on extending this to a general (multi-task) case, but it’s more complex
c
awesome! thanks so much for looking into it
I was going to try out manual looping with the "imperative api" (instead of the context manager style I'm currently using)... but can you confirm that that the outer Flow object would not contain the task return values anywhere?
according to this proposal description it sounds like the only option is to do the looping inside the task
j
I suspect that this proposal would be the right framework for your use case, as you would be able to use the loop to discover and return results -- I'm not entirely sure what your objectives withe the imperative API is, though, as it has all the same control flow structures as the functional one. Let's see if we get this proposal implemented in the near term and work to meet your use case
🎉 1
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by