Can I horizontally scale prefect workers? Reading the code, it seems that the worker fetches the entire list of schedulable flows and submits them. I don't see any kind of locking mechanism. Is the only way by assigning 1 work queue per worker? https://github.com/PrefectHQ/prefect/blob/7294b5b0aee637f9e7f699c91958aec417a87aa2/src/prefect/workers/base.py#L915-L921

hi @Lev Zemlyanov - you can, there are server orchestration rules that would abort flow runs if there was a race condition like two workers picking up the same flow run, since a flow run cannot transition from RUNNING -> RUNNING you might be interesting in this, which was made with horizontal scaling in mind and based on websocket subscriptions instead of polling

@Nate the flow run would be aborted? Do you mean the entire flow run would be aborted or just the "duplicate" one on the second worker?