I managed to deploy to ECS a task that runs a local agent. The problem is that this agent is not able to run flows stored with Docker. I tried to chagne the agent on ECS to docker but it doesn't run. Is it possible to deploy and agent that runs docker on ECS? I don't want just to execute the flows on ECS. I need the agent itself to be running on ECS.

Hi @Levi Leal —

prefect agent docker start

fails? Presumably this is because ECS is running a docker container and then you are trying to run docker within that container which is ‘docker-in-docker’ which is possible but not generally a good pattern (and not setup in the

prefecthq

images)

@Zanie, is there any recommendation on how to run the agent on ECS?

I typical pattern is to run docker containers on ECS itself (ie using the ECSAgent) rather than within an already running task. You can certainly get docker-in-docker working using a custom image if you want though, see https://hub.docker.com/_/docker/

I'l' explain our scenario so maybe it makes more sense. We have prefect server already running on EKS. We have a K8s Agent there and it's working fine. We have a set of applications, including our database, running on a private VPC and we want to get data from this database on prefect. Now we need to create an Agent that will run on this private VPC so that we have access to the database. When I run

prefect agent ecs start

it "spawns" the agent on my local machine. We need this agent to be on the private VPC. We already identified that it is possible to run a EC2 instance and then start the agent using the cli, but we'd like to avoid using EC2. This is the reason we're trying to use ECS + fargate. Maybe this is the wrong approach, and using EC2 inside the VPC is the best way, we don't know yet. What would you recommend on this situation?

You could run the ECS agent in an ECS container since it’s spawning additional ECS jobs rather than trying to run a docker container locally. You could also run the ECS agent (or Docker agent) in a small EC2 instance which would be easier to get going.

Got it. I think I'll try the EC2. Thanks 😄

You could also get docker-in-docker working but I don’t really see a benefit of ECS over EC2 there.

I had quite the battle with this myself lately. My solution was to build myself a bastion server and another server that is internally private within the same private subnet as the ECS Fargate cluster, as well as an ALB, and a NAT for egress traffic (to pull the containers and to allow the agent to connect to the cloud) On that internal server, I started agents via a container from my own images for starting local agents (presumably you could use any flavor of agent though). Then the agents are configured to pull from GitHub storage and then launch those tasks into the ECS cluster with dark-cloudprovider