@Marvin can I run more than on worker for a work queue in prefect 3.0 without duplicating flow runs

Yes, you can run multiple workers for a work queue in Prefect 3.0 without duplicating flow runs. Prefect workers are designed to poll work pools for new runs, and they manage the execution of these runs. Each worker will only execute a flow run once, even if multiple workers are polling the same work pool. This ensures that flow runs are not duplicated across workers. For more details, you can refer to the Prefect documentation on workers.

@Marvin how do the worker not pick up the same flows ready to run in the queue?

Prefect workers avoid picking up the same flows ready to run in the queue through a combination of work queues and concurrency limits. Each worker polls the work queue and retrieves flow runs that are ready to be executed. The system ensures that once a flow run is claimed by a worker, it is marked as such, preventing other workers from picking it up. The use of priorities and concurrency limits further helps manage the order and distribution of flow runs across available workers. This mechanism ensures that flows are executed in the intended order and that no duplication occurs across workers. For more details, you can refer to the Prefect documentation on work pools.