Hi folks, I've used Google Kubernetes engine a bit in a previous role, but we were running airflow on a VM and just sending jobs off to k8s, so it was pretty different than the parallel compute options we've got with Prefect. I switched to Prefect about a year ago (what a nice change), and I was most recently running a pretty simple setup with just a beefed up EC2 since my light pipelines were nowhere near hitting schedule conflicts. Now I'm looking to move my new company's pipelines over to Prefect and I need parallelization. So, looks like I need to go with ECS or EKS (we're on AWS). I haven't had to set up either of these services before. I guess my main question is: where does the agent actually sit? I get the config files etc., but this'll be my first time not running the agent on a VM, and the VM route is so simple because I just go into the machine and start the agent from the terminal. Where does the agent get initialized in the case of ECS or EKS?

ECS agent can sit anywhere as long as it’s authenticated to submit jobs. I could have one on my laptop. Kubernetes agent normally lives as a pod in the cluster.

Thanks! And in your opinion, what's the easiest option to get up and running?

If your org happens to already have a cluster and people maintaining it, it’s that. If from scratch, ECS