I'm trying to figure out how to use prefect to make around 25000 api calls but only execute 10 at a time. I know similar questions have been asked before but I can't figure out the right way how to do it so any help would be greatly appreciated! What i've tried so far: • The suggestion that i found here on slack using semaphore, this only works locally • Using custom code to limit conucrrency using asyncio • Previous options but with async definitions: https://docs.prefect.io/tutorials/execution/#asynchronous-execution Other questions I have: • Should i make 25000 tasks? • Should i have one task that makes all the calls? Appreciate the help!

I’m about to embark on doing something very similar and intrigued with how you’ve implemented this? How have you got on?

I didn't get any guidance from here or the docs. In the end i ended up using a Threadpool which worked for my usecase but I'm not sure if theres a better way to do things.

Threadpool also sounds like what I was going to do. Did you end up making all the api calls seperate tasks? How did the UI feel about this if so?

I tried with separate tasks at first but this broke Prefect. In the end i used a single task.

how about the task concurrency limit? Or you more worried about not creating all the tasks up front? I recently did something where I would create all the tasks in chunks, so you have 1000 tasks, call

.submit()

on the first chunk of 100 and wait for them to all finish before processing the next etc… not sure how efficient it is but it did finally get my flow to finish successfully