Hi Team, I want to set up an environment with fast responding, and a cron scheduled flow should be executed right after the planned time point. Meanwhile the load maybe high sometimes, such as tens of flows to trigger in a second. Do you think Prefect can achieve the requirement? Or any suggestions about the deployment and settings? I want to just use local agent to execute the flows, of course there could be more than one agent instances. Thanks

So I believe that yes it can, but for server specifically, you need to configure Prefect Server to handle more than 10 runs at the same time. See this

Thanks @Kevin Kho, in this pull request, it mentions “when a single flow has more than 10 runs scheduled at exactly the same time”. To my case, I will have different flows, and tens of different flows could be triggered at the same time. So does this pull request still apply?

I don’t expect it to, but if you do run into issues where some flows are being lost, this is the place to start for sure

OK. Another thing is that seems scheduler has some latency to do the scheduling, seems it sleeps for some interval:

{"severity": "DEBUG", "name": "prefect-server.Scheduler", "message": "Sleeping for 60.0 seconds..."}
{"severity": "DEBUG", "name": "prefect-server.ZombieKiller", "message": "Sleeping for 120.0 seconds..."}

Could it be the bottle neck of fast responding?

Will look a bit

OK, then that should be fine. But the interval should be configurable, right?

So it’s not immediately configurable. You can find it here. That

Sleeping for 60 seconds

I believe is the gap between the end of the current loop and the next loop. (It will run again in 60 seconds). So if the default is 150 seconds, I guess it took 90 seconds to run.

If it’s just the latency for newly created flow to be found by the scheduler, that should be fine.

Oh it wouldn’t work with Prefect. Prefect is primarily a batch oriented tool….so you could interact with a task queue from Prefect using Python. For example, AWS has SQS which is just a big queue of stuff to process that feeds into Lambda. You can add to the queue and stuff with Prefect.

I see. Seems the latency on agent side is fine, 0.25 second (https://github.com/PrefectHQ/server/blob/938df638465cc920756cbd769f821f02233fd468/src/prefect_server/services/agents/local_agent.py). So what I need to do is to have enough agent threads to execute the task.

Sure! The tricky thing might be the lack of a load balancing for the agents, but we’ll see

You mean one agent could grab all the flow runs even I run multiple agent instances?

Yes because they just poll on a cycle so it’s just a matter of which one picks it up first.

I can try that. Thanks for the reminder!