Is it possible to yield results from a task into another task? I have a task that pulls data from an API in pages, and rather than accumulating ALL of the data into a list before passing it to the next task, it would be much more memory efficient to yield each page at a time.

You can’t

yield

a task output, or it doesn’t really help because results are held in memory. I assume you know of mapping but it doesn’t fit your use case either right?

I don't think mapping could be used here. Here is some pseudo code

@task
def get_api_data():
    items = []
    for r in stripe.BalanceTransaction.list(limit=100).auto_paging_iter():
        items.append(r)
    return items

@task
def insert_to_db(records):
    # logic to insert into db here


with Flow("test") as flow:
    items = get_api_data()
    insert_to_db(items)

Lets pretend that

get_api_data

returns 10's of thousands of records. All of this would have to be held in memory before passing it to the insert into db task.

Yeah that’s right. This can’t be done. If might be friendlier in 2.0 though because you can do a for loop and run those of these tasks in the for loop with different offsets