How would you deal with the following task. you get a csv file with 10000 locations and want to get the weather for each one of them. (given there is o batch api) so every location would be another prefect task ? or would you structure it in batches to make the number of tasks smaller, but then the caching and retry want work that great ?

Up to you. If you need to make 10,000 API calls you could use a task runner to do it async. We’ll see that tomorrow. https://docs.prefect.io/v3/develop/task-runners You could use

.map

to iterate. https://docs.prefect.io/v3/develop/task-runners#mapping-over-iterables

so there is no issue in having 1000 tasks. there will be lots of arrows in the ui 🙂

We can handle lots of arrows.