Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
j
James Phoenix
07/29/2022, 1:24 PMWill prefect 2.0 be supported via kubernetes GKE auto-pilot?
j
Joshua Greenhalgh
07/29/2022, 1:27 PMI hope soo! I can't imagine there would be anything that could mean it is not? What are you concerned about?
j
James Phoenix
07/29/2022, 1:29 PMI am new to prefect and just wanted to know if it is possible 😉
I’ll try auto-pilot first I think 😛
j
Joshua Greenhalgh
07/29/2022, 1:38 PMI think in most cases they are interchangeable - the main diff is that google is managing the autoscaling - but the k8s apis are exactly(ish) the same I think
r
Rajvir Jhawar
07/29/2022, 2:08 PMI deploy on GKE autopilot and have no problems
j
James Phoenix
07/29/2022, 2:09 PMPerfect that’s good to know, thanks @Rajvir Jhawar
When I deploy my flows into buckets, how do I attach additional dependencies? @Rajvir Jhawar?
r
Rajvir Jhawar
07/29/2022, 2:11 PM@James Phoenix The only issue you encounter with GKE autopilot is the cost associated with running an agent 24/7. GKE autopilot bills per hour your pod is running. The great thing about prefect is you can run you agent outside your cluster (you lose scaling + auto healing) but the price difference might be worth it
I run my agent on a seperate VM (make sure you have kubectl installed on said VM and then connect it to your cluster)
@James Phoenix i not quite sure i understand your question. Do you mean third party libraries?
Then you would package them up in a docker image and deploy to CR (container register).
There are others way of doing it, depends on your devops setup
j
James Phoenix
07/29/2022, 2:14 PMAh gotcha, so if I am running a VM it is cheaper than K8s for the deployment? @Rajvir Jhawar Perhaps I should use a standard k8s setup then with a lower number of resources? I only want this for my own personal usage.
Gotcha, so use docker images via Github container registry / Docker hub
r
Rajvir Jhawar
07/29/2022, 2:18 PMGKE autopilot is billed basically by how long you run a pod for:
So what i do to save some money is have my k8 jobs run in the cluster, but then have the agent in a separate VM constantly polling the prefect api then it would send the requests to your cluster.
That way your not paying for the price of an agent running 24/7.
Now you go the route of using a standard k8 cluster that also up to you.
I prefer autopilot for ease of setup. Also if it fits into your budget just deploy your agent there as well.
Yea whatever docker container register you want to use is going up to you. As long as prefect knows where your images are stored you are good