@Marvin how does work get scheduled if jobs are marked as late? Will scheduled time still be used?

In Prefect 3.x, if a job is marked as "Late," it means that the job's scheduled time has already passed. However, the scheduled time is still relevant and used. When a task or flow run is marked as late, it transitions to a "Late" state but will still execute as soon as possible. The scheduling mechanism will check the scheduled time, and if it's in the past, it will execute the task immediately. This is evident from the logic that calculates the sleep time until the scheduled time, and if the scheduled time is in the past, it proceeds with execution without waiting. For more details, you can refer to the Prefect GitHub source code and task engine implementation.

@Marvin So if 2 flows have the same priority and schedule time, is there any guarantee to which executes first

In Prefect 3.x, when two flows have the same priority and are scheduled for the same time, there is no guaranteed order for which flow will execute first. Priority determines the order of flow runs submitted for execution, but if two flows have the same priority and are not limited by concurrency, they may be submitted in any order. The execution order is primarily influenced by priority levels, and when those are identical, other factors like concurrency limits or the internal scheduling mechanism may influence the order. However, without specific constraints or configurations, the execution order for flows with the same priority and schedule time is not deterministic. For more details, refer to the work pools documentation.