Jeff Kehler
07/06/2022, 3:54 AMKevin Kho
yield
a task output, or it doesn’t really help because results are held in memory. I assume you know of mapping but it doesn’t fit your use case either right?Jeff Kehler
07/06/2022, 4:26 AM@task
def get_api_data():
items = []
for r in stripe.BalanceTransaction.list(limit=100).auto_paging_iter():
items.append(r)
return items
@task
def insert_to_db(records):
# logic to insert into db here
with Flow("test") as flow:
items = get_api_data()
insert_to_db(items)
Lets pretend that get_api_data
returns 10's of thousands of records. All of this would have to be held in memory before passing it to the insert into db task.Kevin Kho