I'm quite confused on how to achieve this primitive thing. 1. Have 4 workers in a pool (A, B, C, D) 2. Created a deployment 3. Submit jobs to run_deployment 4. Have each worker only do a single task Currently for some reason i can get A, A, C ,D, meaning a single worker picks more than one task

hi @Māris - why do you need multiple workers? in general, a worker will listen to a work pool (or specifically, if you want, a work queue within that work pool) there's no "routing" that says a specific worker listening to a pool should pick up a specific run, besides work queue priority so if you have many workers listening to the same pool, to the workers its just a race of who picks up the deployment run first

You could use multiple queues in a work pool, with each worker listening to a specific queue. I suppose the question is what are you wanting to achieve with your scenario?

I have a really heavy cpu workload. A single worker should only run a single job at a time. I cant have multiple queues because then some workers will finish their queues faster than others and now i have to wait for possinly the last queue which can randomly could be the one with the hardest / longest tasks

so an important point is the worker process isn't necessarily "where the work happens" generally speaking, workers are responsible for submitting work, but only in the case of the process worker is that work actually taking place as a subprocess of the worker in the docker or kubernetes worker for example, the worker just submits the flow run as a container according to the protocol for that runtime ie as a docker container or as a kubernetes job so if you have one docker/k8s worker, it can submit N jobs on separate containers, each container having its own mem/cpu requests

I am using docker to execute the flows. Im using my own built image. In that case i want only a single container at all times. When the flow in container is done and exitted, I want to spin up the next container task

Concurrency on the work pool might help? If you have more than one worker talking to a single queue in a work pool though, I don't know of a way to ensure each worker gets 1 and only 1 task at a time.