Hello, I can see a race condition between LateRuns service in prefect server and submission of flow run in worker. In detailed, if flow runs are polled from both worker and LateRuns service in the same second, it might be coming to the fact that both are considered it to state “changeable”. In my case, after flow run is submitted to execution, flow run is switched to PENDING state. At the same time, in prefect server, the background service LateRuns is setting flow run to SCHEDULED state with Late name. As a consequence, worker pick it up again and submit the flow run and cause the second flow run to Fail even though the first flow run is finished properly. Is there any recommendation on this to make worker and server work smoothly together? Or is there a way to restrict to set state of flow run from Pending -> Scheduled? I attached a view on flow run states here for reference.

Hi, i've made this PR here that should hopefully address your problem. It should go out likely in today's release

@Jake Kaplan awesome. Thanks, Jake. How could I test out the new version?

hey it should be included if you install the latest version

prefect==2.16.1

Thank you, @Jake Kaplan 👍