@Marvin when does Prefect server change the status of a run to Late

In Prefect, a run is marked as "Late" when its scheduled start time has passed, but it has not transitioned to "Pending." This transition to "Late" typically occurs 15 seconds after the scheduled start time by default. This status indicates that the run was supposed to start but hasn't yet moved to the next stage, often due to delays or other scheduling issues. For more details, you can check the Prefect documentation on managing states.

@Marvin Will runs in the Late state ever be picked up for execution by the workers or will they stay in that state indefinitely

Runs in the "Late" state can still be picked up for execution by workers. The "Late" state simply indicates that the run's scheduled start time has passed without transitioning to "Pending." Workers, which poll work pools for new runs to execute, will still consider these runs for execution as long as they are not blocked by other conditions or states. For more details on how workers interact with work pools, you can refer to the Prefect documentation on workers and work pools.