I’m trying to decide between cloudrun jobs and k8s as the executor. My concern with cloudrun jobs is that there is a hard limit of 10k concurrent tasks. How does prefect currently handle if more than 10k tasks were loaded onto a cloudrun jobs infrastructure?
✅ 1
m
Mike Grabbe
01/10/2023, 1:33 PM
Im not sure about Prefect's ability to handle 10k concurrent tasks, but I do know you have the ability to throttle concurrent tasks either by setting a work queue concurrency limit or a tag-based task run concurrency limit.
j
James Phoenix
01/10/2023, 2:24 PM
And will the remaining tasks just be pending?
I like the throttling idea, thanks @Mike Grabbe
m
Mike Grabbe
01/10/2023, 2:24 PM
Yes, they should stay in pending status until there's room in the queue
😛anda-dancing: 1
j
James Phoenix
01/10/2023, 2:24 PM
Perfect, thanks 🙂
Really appreciate it
d
Danilo Drobac
01/10/2023, 4:06 PM
@James Phoenix Out of interest, what's your use-case that will potentially run a Flow 10k times concurrently?
j
James Phoenix
01/10/2023, 4:23 PM
A GPT-3 article generator, all of the tasks are quite small so I’m hopefully expecting volume with a small req/compute size.