With prefect server and localagent is there any way to limit job execution concurrency? If I get 1000 jobs submitted in one min, but that server can really only handle executing 150 at a time before it falls over, what ways do I have of improving this situation?

Hey @Adam Brusselback, unfortunately not. It was explored here and was not performant so the effort was stopped.

These are fast executing tasks, so the server can easily get through all 1000 in a full min, but if it tries launching all at once, progress stops. If it launches a reasonable number of them, waits for them to finish, then launches more, we get through that queue of 1000 launched flows nice and quick

It sounds like you can rearchitect this then to be controlled by a “main flow” using LocalDaskExecutor to treat it like a queue?

Maybe, these are individual jobs running for each client in my saas. They can be on the default schedule, or an custom schedule

I am honestly not seeing another way other than starting up another agent (with different compute backing it). Or Cloud cuz Cloud has flow run concurrency limiting and task run concurrency limiting but these need orchestration background services that are not shipped with server.

Yeah, that was essentially what I was trying to use Prefect to replace, my current job scheduler infrastructure handles these small tasks fine

The lower-bound of Prefect jobs where it’s still performant is around 1 min or 30 seconds because it’s primarily a batch orchestrator and there is some overhead with the API states.

Yeah, a lot of these jobs are going to run for <1 second if it turns out there is nothing to do (the majority of the time), and then will take ~20s-2min if there is work to do

LocalDaskExecutor is a multiprocessing pool so it’s the “do not use more than X threads/processes to execute the tasks”.