How does the scheduling component of Prefect actually work? I get that the agent asks for work, but how does the API know which flows should be ran? I am guessing it has the next N scheduled times computed, but is that pre computed on some schedule? Or calculated on API request?

Hey @Lawrence Finn, when you have a schedule attached to the Flow run, it creates the scheduled flow runs already. (This means when you change the default parameters, you need to toggle the schedule on and off to get the future runs with the new parameters. ) The API queries to see if there are any scheduled flows to be run soon. I don’t think it has anything to do with Towel. Have you seen this part of the docs?

Ah yeah, I was looking in the wrong repo 😛 https://github.com/PrefectHQ/server/blob/master/src/prefect_server/services/towel/scheduler.py

Yeah is it a race condition that happens when you have two agents polling at a close interval. This is a bit beyond me, but what I can say is that we have code that explicitly handles this in Cloud (and I think in Server as well), but I have seen this issue crop up for a Server user recently. It is more robustly handled in Cloud. Really though you don’t need multiple agents for the same machine because one agent just does the same work as two and adding more doesn’t add anything. If your agents live on different machines, then we recommend distinct labels otherwise you can run into these race conditions.

Makes sense, we are probably going to use the cloud offering anyway 😄. Thanks for the info

See this and this for more info.