https://prefect.io logo
Title
m

madhav

02/21/2022, 4:56 PM
Hi all - we are testing out production deployment strategies for our prefect agents. We have read: [1][2][3]. Currently we are trying to benchmark, test, and determine costs for deploying to ECS + Fargate, GKE with autopilot, and single large VM with a local executor. Does anybody have rough comparisons by any chance? Cost of course will be sensitive to our specific workflow - but was curious if anyone has a good idea of how the costs approximately compare? [1] https://gist.github.com/palewire/072513a9940478370697323c0d15c6ec, [2] https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a, [3} https://rdrn.me/scaling-out-prefect/.
a

Anna Geller

02/21/2022, 5:00 PM
ECS Fargate is probably the way to go, since you pay only for containers that you spin up, not for VMs. with Vertex you pay per machine and GKE with autopilot similarly since you need to have Kubernetes control plane with all components running at all times
m

madhav

02/21/2022, 5:03 PM
got it, that makes sense. thank you
👍 1
is there a recommended guide for running flows on ECS+Fargate?
a

Anna Geller

02/21/2022, 5:57 PM
I wrote a blog post about it here. But just FYI using FARGATE_SPOT doesn’t work atm, there is an open issue for it. But if you use FARGATE launch type, the setup from the blog post should work.
m

madhav

02/21/2022, 5:58 PM
excellent, thank you. will take a look.