Hi all! My team is considering a switch to Google ...
# prefect-communityt
Thet Naing
02/28/2023, 2:53 PMHi all! My team is considering a switch to Google Kubernetes Engine for our Prefect infrastructure. Does anyone have recipes or examples of deploying flows with GKE clusters in Prefect 2.x? Any help would be greatly appreciated.
c
Christopher Boyd
02/28/2023, 3:03 PMUsing a pipeline or ..? The process is identical across any cloud since you are just building a deployment and applying it.
The only change will be to your storage and credentials more than likely.
If you use a kubernetes-job for your infrastructure block, and the agent runs in GKE then it will pull down the flow runs appropriately
t
Thet Naing
02/28/2023, 4:10 PMI have an agent running in GKE and picking up flow runs when using the base Prefect image, but can't seem to get it to work with my custom image hosted in GCP. Was thinking it'd be helpful to see some examples end to end in case I did something wrong in the process
Thet Naing
02/28/2023, 4:22 PMBtw, thanks for your post in Discourse! Super helpful for getting even this far 🙂
c
Christopher Boyd
02/28/2023, 4:22 PMI have a working pipeline that looks like this:
Copy code
image = "<http://gcr.io/|gcr.io/>" + GCP_PROJECT_ID + "/" + environ['PROJECT_NAME']
k8s_job = KubernetesJob(
image=image,
namespace="prefect2",
# name="healthcheck",
name=environ['PROJECT_NAME'],
customizations=customizations,
env=dict(
GCP_PROJECT_ID=GCP_PROJECT_ID,
GCP_RESULTS_BUCKET=GCP_RESULTS_BUCKET,
PREFECT_VERSION=PREFECT_VERSION,
PYTHON_VERSION=PYTHON_VERSION
),
labels={"environment": f'{APP_ENVIRONMENT}'.lower()},
finished_job_ttl=600,
job_watch_timeout_seconds=600,
pod_watch_timeout_seconds=600
)
deployment = Deployment(
name=f"flow-{APP_ENVIRONMENT}",
flow_name="flow-{APP_ENVIRONMENT}",
version=1,
work_queue_name="dev",
infrastructure=k8s_job,
path="/opt/prefect/flows",
parameters=params,
entrypoint="flow.py:main"
)Christopher Boyd
02/28/2023, 4:23 PMthe ability to pull from your container registry would be an iam policy permission on the GKE cluster itself though
Christopher Boyd
02/28/2023, 4:24 PMbut as long as you specify the right image, and have the permissions. it’s fairly straightforward (that is, not cloud specific)
❤️ 1
t
Thet Naing
02/28/2023, 4:24 PMI previously used my docker image for deploying an agent to a GCP Compute Engine VM, so I have an entrypoint at the bottom of the Dockerfile. Do you think that could be causing issues?
c
Christopher Boyd
02/28/2023, 4:25 PMthe flow will override your docker entrypoint
t
Thet Naing
02/28/2023, 4:26 PMI see
Thet Naing
02/28/2023, 4:28 PMThe thing that's tripping me up is:
• If I don't specify the image, I can run flows with standard dependencies that are included in the Prefect base image, which tells me it's properly connected to the cluster's compute
• If I do specify the image, my flows stay in
Pendingn
Nate
02/28/2023, 4:36 PMhmm I'd be curious to see agent logs, sounds like it could potentially be a permissions thing with pulling the image from your registry?
t
Thet Naing
02/28/2023, 4:47 PMSeems to be picking up the flow runs fine, haven't seen any indication of errors with pulling down the image. I imagine that issue would come up prior to the flows being picked up, right?
Thet Naing
02/28/2023, 4:51 PMIt's failing due to a
BackoffLimitExceededc
Christopher Boyd
02/28/2023, 5:31 PMHow did you define the deployment? BackoffLimit says it seems like the pods are in fact starting , and failing / dying. You should have logs for the failed pods indicating why they failed
t
Thet Naing
02/28/2023, 8:43 PMThanks for all the help! I think it was an issue with my Dockerfile. I managed to get it up and running by removing the entrypoint from the Dockerfile
👍 1