https://prefect.io logo
Channels
Powered by
j

Joe Hamman

08/10/2021, 3:35 PM
Hi folks - we’re setting up a new prefect agent in our Kubernetes cluster on Google Cloud and I could use some pointers getting the correct permissions configured. I’m starting with this:
Copy code
prefect agent kubernetes install -k $KEY --namespace=staging --rbac | kubectl apply --namespace=staging -f -
I then submit a flow that uses the 
KubernetesRun
config and 
GCS
storage configured as:
Copy code
run_config = KubernetesRun(cpu_request=2, memory_request="2Gi", image='<http://gcr.io/carbonplan/hub-notebook:c89f7f1|gcr.io/carbonplan/hub-notebook:c89f7f1>', env={'TZ': 'UTC'})
storage = GCS("carbonplan-scratch", project='carbonplan')
This results in an error message like this:
Copy code
└── 23:15:59 | INFO    | Submitted for execution: Job prefect-job-d8a8a648
└── 23:16:05 | INFO    | Entered state <Failed>: Failed to load and execute Flow's environment: Forbidden('GET <https://storage.googleapis.com/storage/v1/b/carbonplan-scratch?projection=noAcl&prettyPrint=false>: Caller does not have storage.buckets.get access to the Google Cloud Storage bucket.')
So, I gather my agent doesn’t have the correct IAM privileges to read the flow from GCS. Next I tried adding a service account to my agent:
Copy code
prefect agent kubernetes install -k $KEY --namespace=staging --service-account-name pangeo --rbac | kubectl apply --namespace=staging -f -
Here I’m pointing to my kubernetes service account called 
pangeo
which has been given 
storage.objectAdmin
permissions. However this results in the same error as above. So now I’m wondering if I’m missing something more fundamental here. If anyone has suggestions on where to look for more details on setting up prefect on GKE, I’d certainly appreciate it.
k

Kevin Kho

08/10/2021, 3:56 PM
Hey @Joe Hamman, you hunch is right that it seems authentication related. I haven’t worked on stuff like this myself, will ask the team for input.
j

Joe Hamman

08/10/2021, 3:57 PM
Thanks @Kevin Kho!
m

Mariia Kerimova

08/10/2021, 4:18 PM
Hi Joe! Looks like role 
storage.objectAdmin
doesn't have buckets permissions. Found it here: https://cloud.google.com/storage/docs/access-control/iam-roles. Can you test with 
roles/storageAdmin
role, and let us know if it fails again?
j

Joe Hamman

08/10/2021, 8:03 PM
Adding the storageAdmin role seems to have worked
here’s a complete setup that may be useful to others:
Copy code
gcloud iam service-accounts create prefect
	gcloud iam service-accounts add-iam-policy-binding \
		--role roles/iam.workloadIdentityUser \
		--member "serviceAccount:$(PROJECT).svc.id.goog[$(NAMESPACE)/prefect]" \
		prefect@$(PROJECT).<http://iam.gserviceaccount.com|iam.gserviceaccount.com>
	
	kubectl create serviceaccount prefect --namespace $(NAMESPACE)
	kubectl annotate serviceaccount \
		--overwrite --namespace $(NAMESPACE) \
		prefect \
		<http://iam.gke.io/gcp-service-account=prefect@$(PROJECT).iam.gserviceaccount.com|iam.gke.io/gcp-service-account=prefect@$(PROJECT).iam.gserviceaccount.com>
	gcloud projects add-iam-policy-binding $(PROJECT) \
		--member serviceAccount:prefect@$(PROJECT).<http://iam.gserviceaccount.com|iam.gserviceaccount.com> \
		--role roles/serviceusage.serviceUsageConsumer
	gsutil iam ch serviceAccount:prefect@$(PROJECT).<http://iam.gserviceaccount.com:roles/storage.admin,objectViewer|iam.gserviceaccount.com:roles/storage.admin,objectViewer> <gs://carbonplan-prefect>
	gsutil iam ch serviceAccount:prefect@$(PROJECT).<http://iam.gserviceaccount.com:roles/storage.admin,objectViewer|iam.gserviceaccount.com:roles/storage.admin,objectViewer> <gs://carbonplan-scratch>
	
	prefect agent kubernetes install -t $(KEY) --rbac --namespace=$(NAMESPACE) --service-account-name prefect --image-pull-policy=Always | kubectl apply --namespace=$(NAMESPACE) -f -
k

Kevin Kho

08/10/2021, 8:11 PM
@Marvin archive “storage.objectAdmin in Google Cloud Platform does not have bucket permissions”
m

Marvin

08/10/2021, 8:12 PM
Powered by