Thread
#best-practices-coordination-plane
    Jacob Bedard

    Jacob Bedard

    3 months ago
    Hi folks, I've used Google Kubernetes engine a bit in a previous role, but we were running airflow on a VM and just sending jobs off to k8s, so it was pretty different than the parallel compute options we've got with Prefect. I switched to Prefect about a year ago (what a nice change), and I was most recently running a pretty simple setup with just a beefed up EC2 since my light pipelines were nowhere near hitting schedule conflicts. Now I'm looking to move my new company's pipelines over to Prefect and I need parallelization. So, looks like I need to go with ECS or EKS (we're on AWS). I haven't had to set up either of these services before. I guess my main question is: where does the agent actually sit? I get the config files etc., but this'll be my first time not running the agent on a VM, and the VM route is so simple because I just go into the machine and start the agent from the terminal. Where does the agent get initialized in the case of ECS or EKS?
    Kevin Kho

    Kevin Kho

    3 months ago
    ECS agent can sit anywhere as long as it’s authenticated to submit jobs. I could have one on my laptop. Kubernetes agent normally lives as a pod in the cluster.
    Jacob Bedard

    Jacob Bedard

    3 months ago
    Thanks! And in your opinion, what's the easiest option to get up and running?
    Between Kubernetes and ECS, that is
    Kevin Kho

    Kevin Kho

    3 months ago
    If your org happens to already have a cluster and people maintaining it, it’s that. If from scratch, ECS