Hey there, is there a functional limit to the numb...
# ask-community
s
Hey there, is there a functional limit to the number of tasks that can be in the
Pending
state at once, specifically with regard to Prefect Cloud API Rate Limits? I have a flow that operates on many files, with 1 task per file. We limit to 4 tasks running at once by using tag-based concurrency limits. But there is a steady stream of new tasks added to the
Pending
queue, about one task every 2 seconds. Eventually I get hit with
429 Too Many Requests
and everything enters the
Crashed
state. The tasks are relatively long-running (minutes), so there's not some flurry of activity there that could overwhelm the API. Logging is also very minimal, so that isn't triggering the limit either. The endpoint that complains changes; usually it's
/set_state
, but sometimes
/task_runs
. Maybe there's some API operation that happens for every
Pending
task, and once there are enough then everything breaks?
There are also no other flows running at the same time, just this one I'm working on