Ofek Katriel
06/20/2022, 8:48 PMbuild-and-deploy-production:
stage: build
# image: prefecthq/prefect:latest-python3.9
variables:
DOCKER_HOST: <tcp://localhost:2375>
DOCKER_TLS_CERTDIR: ""
# image: python:3.9.13-buster
image: google/cloud-sdk
services:
- docker:19.03.1-dind
script:
- pip3 install prefect
- echo $PREFECT_PRODUCTION_SERVICE_ACCOUNT > /tmp/$CI_PIPELINE_ID.json
- gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
- gcloud config set project my project
# - gcloud components install docker-credential-gcr
- gcloud auth configure-docker --quiet
- prefect auth login -k mytoken
- cd ./asin2vec_pipeline/asin2vec
- pip3 install -r requirements.txt
- export GOOGLE_APPLICATION_CREDENTIALS=/tmp/$CI_PIPELINE_ID.json
- prefect register --project "onboarding" --path flow.py --name "asin2vec_v2"
as you can see, tried with images: prefecthq/prefect:latest-python3.9, cloudsdk, and python.
with all had the same error: InterruptedError: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myproject/locations/us-central1/repositories/data-pipelines" (or it may not exist)
to be clear, locally it works, when I authenticate with this service account.Anna Geller
06/20/2022, 9:47 PMOfek Katriel
06/20/2022, 10:16 PMflow.run_config = VertexRun(machine_type='n2-highcpu-80', labels=["ml"],
service_account=PREFECT_SERVICE_ACCOUNT_NAME)
and that's how the flow config looks like
with Flow(f'asin2vec_v2',
storage=Docker(registry_url=DATA_PIPELINES_REGISTRY_URL,
dockerfile="./Dockerfile"), executor=LocalDaskExecutor(scheduler="processes")) as flow:
Anna Geller
06/21/2022, 12:03 AMOfek Katriel
06/21/2022, 8:35 AMAnna Geller
06/21/2022, 11:28 AM# Copy and paste the contents of the Service Account JSON file into the secret <https://github.com/marketplace/actions/authenticate-to-google-cloud#authenticating-via-service-account-key-json-1>
- name: Authenticate to GCP
uses: 'google-github-actions/auth@v0.7.1'
with:
credentials_json: '${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}'
create_credentials_file: true
Ofek Katriel
06/21/2022, 11:55 AMAnna Geller
06/21/2022, 12:04 PMOfek Katriel
06/21/2022, 1:59 PM