https://prefect.io logo
Title
j

James Phoenix

01/10/2023, 1:00 PM
I’m trying to decide between cloudrun jobs and k8s as the executor. My concern with cloudrun jobs is that there is a hard limit of 10k concurrent tasks. How does prefect currently handle if more than 10k tasks were loaded onto a cloudrun jobs infrastructure?
1
m

Mike Grabbe

01/10/2023, 1:33 PM
Im not sure about Prefect's ability to handle 10k concurrent tasks, but I do know you have the ability to throttle concurrent tasks either by setting a work queue concurrency limit or a tag-based task run concurrency limit.
j

James Phoenix

01/10/2023, 2:24 PM
And will the remaining tasks just be pending? I like the throttling idea, thanks @Mike Grabbe
m

Mike Grabbe

01/10/2023, 2:24 PM
Yes, they should stay in pending status until there's room in the queue
😛anda-dancing: 1
j

James Phoenix

01/10/2023, 2:24 PM
Perfect, thanks 🙂 Really appreciate it
d

Danilo Drobac

01/10/2023, 4:06 PM
@James Phoenix Out of interest, what's your use-case that will potentially run a Flow 10k times concurrently?
j

James Phoenix

01/10/2023, 4:23 PM
A GPT-3 article generator, all of the tasks are quite small so I’m hopefully expecting volume with a small req/compute size.