Steve Suway
05/28/2024, 10:17 PMPending
state at once, specifically with regard to Prefect Cloud API Rate Limits? I have a flow that operates on many files, with 1 task per file. We limit to 4 tasks running at once by using tag-based concurrency limits. But there is a steady stream of new tasks added to the Pending
queue, about one task every 2 seconds. Eventually I get hit with 429 Too Many Requests
and everything enters the Crashed
state. The tasks are relatively long-running (minutes), so there's not some flurry of activity there that could overwhelm the API. Logging is also very minimal, so that isn't triggering the limit either. The endpoint that complains changes; usually it's /set_state
, but sometimes /task_runs
. Maybe there's some API operation that happens for every Pending
task, and once there are enough then everything breaks?Steve Suway
05/28/2024, 10:22 PM