Florian Guily
05/18/2022, 10:16 AMAnna Geller
05/18/2022, 10:36 AMwhere the kubernetes agent is running ?Thanks for reading it and sorry to hear it's not clear where the agent is running! Perhaps I could have made it clearer in the article, so thanks for pointing that out. The agent is running on the AWS EKS Kubernetes cluster as an independent deployment - this post shows how to spin up a 1-node cluster and then how to generate a deployment manifest which then deploys the agent pod - this way, the agent runs on Kubernetes, and thanks to the KubernetesRun run config with the same label as that agent, the flow runs also end up running on the same Kubernetes cluster as the agent.
i suppose this is the only thing that needs to run all the time100% correct yes, the agent is the only process that needs to be running 24/7 when using Prefect Cloud re scalability, there are two options on AWS EKS: 1. fully managed serverless data plane with Fargate - takes care of all the management pain and scale, but as a drawback, it adds quite a bit of latency due to serverless (needs to provision the actual infra) 2. autoscaling policy which is nicely described here
Florian Guily
05/18/2022, 11:34 AMAnna Geller
05/18/2022, 12:11 PMprefect agent xxx start --label zzz
and then this supervisor process could be started on VM boot using crontab:
echo "@reboot root supervisord -c /home/ec2user/supervisord.conf -l /home/ec2user/supervisord.log -u root" >> /etc/crontab
prefect agent kubernetes start --label k8s
then all flow runs spun up by this agent process running on EC2 will be executed as K8s jobsFlorian Guily
05/18/2022, 12:19 PMAnna Geller
05/18/2022, 12:28 PMFlorian Guily
05/18/2022, 12:29 PM