Hi guys, after 2 days of trying to register my flow via gitlabCI I gave up, please help! here's my ci:

build-and-deploy-production:
   stage: build
   # image: prefecthq/prefect:latest-python3.9
   variables:
     DOCKER_HOST: <tcp://localhost:2375>
     DOCKER_TLS_CERTDIR: ""
   # image: python:3.9.13-buster
   image: google/cloud-sdk
   services:
     - docker:19.03.1-dind
   script:
     - pip3 install prefect
     - echo $PREFECT_PRODUCTION_SERVICE_ACCOUNT > /tmp/$CI_PIPELINE_ID.json
     - gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
     - gcloud config set project my project
     # - gcloud components install docker-credential-gcr
     - gcloud auth configure-docker --quiet
     - prefect auth login -k mytoken
     - cd ./asin2vec_pipeline/asin2vec
     - pip3 install -r requirements.txt
     - export GOOGLE_APPLICATION_CREDENTIALS=/tmp/$CI_PIPELINE_ID.json
     - prefect register --project "onboarding" --path flow.py --name "asin2vec_v2"

as you can see, tried with images: prefecthq/prefect:latest-python3.9, cloudsdk, and python.

with all had the same error:       InterruptedError: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myproject/locations/us-central1/repositories/data-pipelines" (or it may not exist)

to be clear, locally it works, when I authenticate with this service account.

Sorry to hear you struggle with this CI. Could you share a walkthrough of which steps did you take and what was the result of each attempt (e.g. as a GitHub gist)? often, it's really helpful to write it all down With GitLab, we've seen some users struggling with setting up permissions - here is how you can set that up and test it locally For Google Artifact registry check this CI example (or in the worst case perhaps you can switch to GitHub Actions then?)

I'm not quite sure I understand your question, within the flow itself I try to deploy it to vertexAI

flow.run_config = VertexRun(machine_type='n2-highcpu-80', labels=["ml"],
                            service_account=PREFECT_SERVICE_ACCOUNT_NAME)

and that's how the flow config looks like

with Flow(f'asin2vec_v2',
          storage=Docker(registry_url=DATA_PIPELINES_REGISTRY_URL,
                         dockerfile="./Dockerfile"), executor=LocalDaskExecutor(scheduler="processes")) as flow:

I'm trying to collect more information in order to figure out what's missing in your setup - currently I don't have enough info to say what's wrong The error you got indicates a permission issue with the Artifact registry - did you check whether your agent has permission to pull that Docker image? Usually you would need to do something like: docker login -u username -p password, and then start a Docker agent so that this agent is authenticated to pull the image from your registry

I’ll clarify - Im trying to register flows via CI-CD and not but running “prefect register” from my own mac. When running “prefect register” with service account X - it all works and the flow gets registered. When doing the same thing in the Ci - I get this permission error. Hope its clear now

ok, so your CI is not authenticated to push the image to the registry - if you're on GitHub Actions, try adding this step before pushing the image:

# Copy and paste the contents of the Service Account JSON file into the secret <https://github.com/marketplace/actions/authenticate-to-google-cloud#authenticating-via-service-account-key-json-1>
      - name: Authenticate to GCP
        uses: 'google-github-actions/auth@v0.7.1'
        with:
          credentials_json: '${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}'
          create_credentials_file: true

so that’s the thing, I’m on gitlab, and I can’t find this example for gitlab, and I literally tried everything

I think this is something you can perhaps raise in GCP Slack? https://googlecloud-community.slack.com/

haven’t you got an example for prefect register via gitlabCI?