Hi guys, another question here, my company has this strict policy of only using containers in cloud VM, and prohibiting the installation of any libraries directly on the local machine. I would like to know if running Prefect client in containers is possible? Specifically, I would like to run DockerAgent in a container, and the registration of my flows should also done in a container. I’ve seen previous Slack discussion on how running DockerAgent in a container will cause issues with memory (as flows containers are created within the agent container). May I know if there is any fix for that? Are there any other downsides to running Prefect client in container?

In general, Docker agent is supposed to run in a local process (not in a docker container) and this local process is a layer between Prefect backend and a Docker Daemon. This agent polls the API for new flow runs, and if there are new flow runs scheduled to run, it then creates new flow runs and deploys those as Docker containers on the same machine as the agent. When Docker agent is running within a container itself (rather than a local process), your flow runs end up deployed as container, but not as individual containers, but rather within the agent container. You effectively have a single agent container spinning up new containers within itself (docker in docker), which may have many unintended consequences. If you want more environment isolation for this agent process, you can run it within a virtual environment. And if you’re super strict that every process must run in a container, check out the KubernetesAgent instead.

@Anna Geller if we use Kubernetes agent, can we run it in a container? Will there be any downside in doing that? Also, we will need to have Kubernetes installed on the machine so that KubernetesAgent can use it right?

yes, correct. All Kubernetes jobs run within pods and within those pods you run containers. There are definitely no specific downsides using KubernetesAgent as opposed to a DockerAgent, in fact, it actually has some additional advantages in terms of scalability etc. But it has some added complexity as well, so as with everything, it’s a tradeoff you need to consider. Yes, you would install it as shown here https://docs.prefect.io/orchestration/agents/kubernetes.html#running-in-cluster

@Anna Geller @Kevin Kho just curious, if we spin up multiple docker agents, can we specify the maximum no of flow job each agent can take?

@Christopher Chong Tau Teng You could do it in Prefect Cloud, this feature is called concurrency limits https://docs.prefect.io/orchestration/flow-runs/concurrency-limits.html