https://prefect.io logo
j

James Phoenix

07/29/2022, 1:24 PM
Will prefect 2.0 be supported via kubernetes GKE auto-pilot?
j

Joshua Greenhalgh

07/29/2022, 1:27 PM
I hope soo! I can't imagine there would be anything that could mean it is not? What are you concerned about?
j

James Phoenix

07/29/2022, 1:29 PM
I am new to prefect and just wanted to know if it is possible 😉
I’ll try auto-pilot first I think 😛
j

Joshua Greenhalgh

07/29/2022, 1:38 PM
I think in most cases they are interchangeable - the main diff is that google is managing the autoscaling - but the k8s apis are exactly(ish) the same I think
r

Rajvir Jhawar

07/29/2022, 2:08 PM
I deploy on GKE autopilot and have no problems
j

James Phoenix

07/29/2022, 2:09 PM
Perfect that’s good to know, thanks @Rajvir Jhawar
When I deploy my flows into buckets, how do I attach additional dependencies? @Rajvir Jhawar?
r

Rajvir Jhawar

07/29/2022, 2:11 PM
@James Phoenix The only issue you encounter with GKE autopilot is the cost associated with running an agent 24/7. GKE autopilot bills per hour your pod is running. The great thing about prefect is you can run you agent outside your cluster (you lose scaling + auto healing) but the price difference might be worth it
I run my agent on a seperate VM (make sure you have kubectl installed on said VM and then connect it to your cluster)
@James Phoenix i not quite sure i understand your question. Do you mean third party libraries? Then you would package them up in a docker image and deploy to CR (container register). There are others way of doing it, depends on your devops setup
j

James Phoenix

07/29/2022, 2:14 PM
Ah gotcha, so if I am running a VM it is cheaper than K8s for the deployment? @Rajvir Jhawar Perhaps I should use a standard k8s setup then with a lower number of resources? I only want this for my own personal usage.
Gotcha, so use docker images via Github container registry / Docker hub
r

Rajvir Jhawar

07/29/2022, 2:18 PM
GKE autopilot is billed basically by how long you run a pod for: So what i do to save some money is have my k8 jobs run in the cluster, but then have the agent in a separate VM constantly polling the prefect api then it would send the requests to your cluster. That way your not paying for the price of an agent running 24/7. Now you go the route of using a standard k8 cluster that also up to you. I prefer autopilot for ease of setup. Also if it fits into your budget just deploy your agent there as well.
Yea whatever docker container register you want to use is going up to you. As long as prefect knows where your images are stored you are good