heyy, so I'm running a task that fetches data from a third party api, returns the response, which is passed to the next task (to index in elasticsearch)... now I need to add pagination support in the first task, and I'm wondering if it's feasible to make the task itself repeat for however many pages are needed, so as not to blow memory or connection limits downstream.. if so could someone recommend the patterns in the docs for this?

If you could write an upstream task which somehow infers how many pages are going to be required in this particular run and produces a list where each element corresponds to a page, you could use task mapping

sweet thanks will check it out

Mapping would provide the benefit of treating each run of the task as a true task, with its own state (individual failures, retries, etc.). It sounds like your alternative is to include all the runtime logic for the loop in a single task. Both are perfectly valid, it just depends on whether you really want each page to be seen by Prefect as a standalone Task or whether treating the full loop as a standalone task is sufficient for your use case

I have a plan for co-opting the mapping mechanism for loops, including while loops, and If you don’t mind I’ll add your use case to the motivation — we’ll keep it on the roadmap