Question: Hey Prefect community, does anyone know if it's possible to have flows scheduled in a reliable and controllable way on entirely different machines through docker workers & workpools? Example: Submit job -> Docker worker on machine 1 pulls job -> creates flow run on machine 1 -> flow creates a subflow -> docker worker on machine 2 pulls job -> creates the subflow on machine 2 -> machine 1 and machine 2 cannot run another flow until they have completed their current flows. Issue: It appears to us that the docker worker will just continually pull from the queue and there's no way to limit the # of flows per machine which means there's absolutely no way to manage system resources. Desired outcome: On prefect-kubernetes we would just specify hard requirements for the machine so that we ensure that each flow is dedicated an entire machine's memory, cpu and gpu. Our use case it that although we have an eks cluster, we are examining leveraging people's machines in office (~200 gpu workstations) for additional compute nightly with minimal overhead... and hoping the answer isn't to install minikube, etc. in addition to going with some on prem k8s option.

you can limit the number of flows an individual worker can manage at any given time with the

--limit

flag on the worker's start command. https://github.com/PrefectHQ/prefect/blob/1bed1976d1e6aef83bb2a1f9b866b7010c3d7c92/src/prefect/cli/worker.py#L88

Thank you so much @Kevin Grismore! I will try this tonight 🙏