Alessandro De Rose
05/15/2023, 5:20 PMprefect deploy --all
to deploy multiple deployments stored in a deployment.yaml
file. I have a build step stored in prefect.yaml
to generate an image containing flow dependencies where push: true
to ensure that the built image is pushed to Google Artifact Registry (from which Cloud Run infra will pull from). The call to deploy
is being carried out in a GitHub action, before calling prefect deploy --all
, I have run gcloud auth configure-docker <http://us-docker.pkg.dev|us-docker.pkg.dev>
to configure gcloud
as the credential helper for docker. However, this action is currently failing with the error unauthorized: access token has insufficient scopes
because Prefect seems to be attempting to push to <http://docker.io|docker.io>
still (The push refers to repository [<http://docker.io/.../...|docker.io/.../...>]
). Any thoughts on what's causing this?Jamie Zieziula
05/15/2023, 8:06 PMAlessandro De Rose
05/15/2023, 8:18 PM<http://docker.io|docker.io>
instead of <http://us-docker.pkg.dev|us-docker.pkg.dev>
. Please see my action steps below:
steps:
- name: Checkout repository
uses: actions/checkout@v3
- id: auth
name: Authenticate to Google Cloud
uses: google-github-actions/auth@v1
with:
workload_identity_provider: ...
service_account: ...
- name: Configure Google Cloud credential helper
run: gcloud auth configure-docker us-docker.pkg.dev
- uses: actions/setup-python@v4
with:
python-version: '3.11'
cache: 'pip'
- name: Install Prefect
run: pip install prefect==2.10.9
- name: Authenticate with Prefect
run: prefect cloud login --key ${{ secrets.PREFECT_API_KEY }} --workspace ${{ secrets.PREFECT_WORKSPACE }}
- name: Deploy all flows listed in deployment.yaml
run: prefect deploy --all
Jamie Zieziula
05/15/2023, 8:20 PMAlessandro De Rose
05/15/2023, 8:26 PM# File for configuring project / deployment build, push and pull steps
# Generic metadata about this project
name: prefect_flows
prefect-version: 2.10.9
# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
requires: prefect-docker>0.2.0
image_name: ###/###
tag: latest
dockerfile: auto
push: true
# push section allows you to manage if and how this project is uploaded to remote locations
push: null
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.git_clone_project:
repository: <https://github.com/###/###.git>
branch: main
access_token: null
Jamie Zieziula
05/15/2023, 8:27 PMbuild:
- prefect_docker.projects.steps.build_docker_image:
image_name: ###/###
starts with <http://us-docker.pkg.dev/|us-docker.pkg.dev/>
Alessandro De Rose
05/15/2023, 8:27 PMJamie Zieziula
05/15/2023, 8:29 PMAlessandro De Rose
05/15/2023, 8:33 PMprefect deployment build
-- this currently creates a new file for each deployment. Is there a flag I can use to get it to append to an existing deployment.yaml file?Jamie Zieziula
05/15/2023, 8:39 PMprojects
you would run prefect project init
when you initially create your project, and then add / update your deployment specs manually rather than via the cliAlessandro De Rose
05/15/2023, 8:41 PMdeployment
approach because I couldn't find much documentation on the required properties. Would you be able to point me in the right direction?
The auto-generated deployment.yaml is rather verbose so if I can get rid of some unnecessary fields that would be great (especially in the credentials/storage sections)Jamie Zieziula
05/15/2023, 8:59 PMAlessandro De Rose
05/15/2023, 9:02 PMJamie Zieziula
05/15/2023, 9:05 PMAlessandro De Rose
05/15/2023, 9:06 PMJamie Zieziula
05/15/2023, 9:07 PMAlessandro De Rose
05/15/2023, 9:15 PMThis deployment doesn't have an associated work queue; runs will be scheduled but won't be picked up by your agents
But the queue is defined in the `deployment.yaml`:
work_queue_name: default
work_pool_name: cloud-run-work-pool
Also within the UI, I can see "Storage Document ID" and "Infrastructure Document ID" are None
. So it doesn't seem to be picking up those from the deployment.yaml
either -- despite it being generated by prefect deployment build
.Jamie Zieziula
05/15/2023, 10:08 PMworker
rather than an agentpull
step in prefect.yaml
— this will replace the need for a storage block // documentAlessandro De Rose
05/15/2023, 10:24 PMJamie Zieziula
05/15/2023, 10:54 PM