Joe Hamman

    Joe Hamman

    1 year ago
    Hi folks - we’re setting up a new prefect agent in our Kubernetes cluster on Google Cloud and I could use some pointers getting the correct permissions configured. I’m starting with this:
    prefect agent kubernetes install -k $KEY --namespace=staging --rbac | kubectl apply --namespace=staging -f -
    I then submit a flow that uses the
    KubernetesRun
    config and
    GCS
    storage configured as:
    run_config = KubernetesRun(cpu_request=2, memory_request="2Gi", image='<http://gcr.io/carbonplan/hub-notebook:c89f7f1|gcr.io/carbonplan/hub-notebook:c89f7f1>', env={'TZ': 'UTC'})
    storage = GCS("carbonplan-scratch", project='carbonplan')
    This results in an error message like this:
    └── 23:15:59 | INFO    | Submitted for execution: Job prefect-job-d8a8a648
    └── 23:16:05 | INFO    | Entered state <Failed>: Failed to load and execute Flow's environment: Forbidden('GET <https://storage.googleapis.com/storage/v1/b/carbonplan-scratch?projection=noAcl&prettyPrint=false>: Caller does not have storage.buckets.get access to the Google Cloud Storage bucket.')
    So, I gather my agent doesn’t have the correct IAM privileges to read the flow from GCS. Next I tried adding a service account to my agent:
    prefect agent kubernetes install -k $KEY --namespace=staging --service-account-name pangeo --rbac | kubectl apply --namespace=staging -f -
    Here I’m pointing to my kubernetes service account called
    pangeo
    which has been given
    storage.objectAdmin
    permissions. However this results in the same error as above. So now I’m wondering if I’m missing something more fundamental here. If anyone has suggestions on where to look for more details on setting up prefect on GKE, I’d certainly appreciate it.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Joe Hamman, you hunch is right that it seems authentication related. I haven’t worked on stuff like this myself, will ask the team for input.
    Joe Hamman

    Joe Hamman

    1 year ago
    Thanks @Kevin Kho!
    m

    Mariia Kerimova

    1 year ago
    Hi Joe! Looks like role
    storage.objectAdmin
    doesn't have buckets permissions. Found it here: https://cloud.google.com/storage/docs/access-control/iam-roles. Can you test with
    roles/storageAdmin
    role, and let us know if it fails again?
    Joe Hamman

    Joe Hamman

    1 year ago
    Adding the storageAdmin role seems to have worked
    here’s a complete setup that may be useful to others:
    gcloud iam service-accounts create prefect
    	gcloud iam service-accounts add-iam-policy-binding \
    		--role roles/iam.workloadIdentityUser \
    		--member "serviceAccount:$(PROJECT).svc.id.goog[$(NAMESPACE)/prefect]" \
    		prefect@$(PROJECT).<http://iam.gserviceaccount.com|iam.gserviceaccount.com>
    	
    	kubectl create serviceaccount prefect --namespace $(NAMESPACE)
    	kubectl annotate serviceaccount \
    		--overwrite --namespace $(NAMESPACE) \
    		prefect \
    		<http://iam.gke.io/gcp-service-account=prefect@$(PROJECT).iam.gserviceaccount.com|iam.gke.io/gcp-service-account=prefect@$(PROJECT).iam.gserviceaccount.com>
    	gcloud projects add-iam-policy-binding $(PROJECT) \
    		--member serviceAccount:prefect@$(PROJECT).<http://iam.gserviceaccount.com|iam.gserviceaccount.com> \
    		--role roles/serviceusage.serviceUsageConsumer
    	gsutil iam ch serviceAccount:prefect@$(PROJECT).<http://iam.gserviceaccount.com:roles/storage.admin,objectViewer|iam.gserviceaccount.com:roles/storage.admin,objectViewer> <gs://carbonplan-prefect>
    	gsutil iam ch serviceAccount:prefect@$(PROJECT).<http://iam.gserviceaccount.com:roles/storage.admin,objectViewer|iam.gserviceaccount.com:roles/storage.admin,objectViewer> <gs://carbonplan-scratch>
    	
    	prefect agent kubernetes install -t $(KEY) --rbac --namespace=$(NAMESPACE) --service-account-name prefect --image-pull-policy=Always | kubectl apply --namespace=$(NAMESPACE) -f -
    Kevin Kho

    Kevin Kho

    1 year ago
    @Marvin archive “storage.objectAdmin in Google Cloud Platform does not have bucket permissions”
    Marvin

    Marvin

    1 year ago