https://prefect.io logo
#best-practices
Title
# best-practices
j

Jacob Bedard

06/01/2022, 2:21 AM
Hi folks, I've used Google Kubernetes engine a bit in a previous role, but we were running airflow on a VM and just sending jobs off to k8s, so it was pretty different than the parallel compute options we've got with Prefect. I switched to Prefect about a year ago (what a nice change), and I was most recently running a pretty simple setup with just a beefed up EC2 since my light pipelines were nowhere near hitting schedule conflicts. Now I'm looking to move my new company's pipelines over to Prefect and I need parallelization. So, looks like I need to go with ECS or EKS (we're on AWS). I haven't had to set up either of these services before. I guess my main question is: where does the agent actually sit? I get the config files etc., but this'll be my first time not running the agent on a VM, and the VM route is so simple because I just go into the machine and start the agent from the terminal. Where does the agent get initialized in the case of ECS or EKS?
k

Kevin Kho

06/01/2022, 2:23 AM
ECS agent can sit anywhere as long as it’s authenticated to submit jobs. I could have one on my laptop. Kubernetes agent normally lives as a pod in the cluster.
j

Jacob Bedard

06/01/2022, 2:34 AM
Thanks! And in your opinion, what's the easiest option to get up and running?
Between Kubernetes and ECS, that is
k

Kevin Kho

06/01/2022, 2:36 AM
If your org happens to already have a cluster and people maintaining it, it’s that. If from scratch, ECS
🙏 1
8 Views