Hi! So I have an external API with pagination, that provides some results with a call and a link to next page of results. Total results (all chunks combined) of this API are huge and do not fit in memory. Ideally I would make a task which pulls that API a generator so that I can start mapping its results before pulling complete. Is it possible to achieve that in Prefect?

Hi @Andrey Tatarinov - you should check out task Looping; I think that'll nicely fit your use case where you want to process results based on your generator some segment at a time but you don't know how many there will be and don't want to store them all in memory by fetching them upstream.

@nicholas so do I understand correctly, that I can do task looping in "generator" task and

.map

it results?

That's correct @Andrey Tatarinov - you could loop that generator task, adding to the results until you're satisfied, and then map over the results you've generated.

Thanks!

In other words you can't pass it between tasks (or iterations of a task?)

yes

In cases like that, you don't need to leave the task or pass the generator to another task, you can always do your looping pythonically and collect the results as normal before passing them along to the next task. You lose a little visibility into where something might break down but not much

collect the results

That works as long as it all fits in memory.

Sorry for the slow reply over here @Andrey Tatarinov - this is definitely doable within a task; one approach i've taken is to loop through the file upstream (python does a great job of reading files line by line or however you need to) to generate a list of indexes. That will make sure you know how much mapping you need to do and you can pass that list downstream to your mapped task and then read the file at those indexes. There are other considerations that I'm sure you're thinking of there but that would allow the file to only be read into memory in ways that are manageable

@nicholas thanks, I think I understand general approach now, will try it out

Let me know how it goes! Happy to help as you have more questions 🙂