Hi there, I would like to use `prefect deploy --al...
# prefect-cloud
Hi there, I would like to use
prefect deploy --all
to deploy multiple deployments stored in a
file. I have a build step stored in
to generate an image containing flow dependencies where
push: true
to ensure that the built image is pushed to Google Artifact Registry (from which Cloud Run infra will pull from). The call to
is being carried out in a GitHub action, before calling
prefect deploy --all
, I have run
gcloud auth configure-docker <http://us-docker.pkg.dev|us-docker.pkg.dev>
to configure
as the credential helper for docker. However, this action is currently failing with the error
unauthorized: access token has insufficient scopes
because Prefect seems to be attempting to push to
still (
The push refers to repository [<http://docker.io/.../...|docker.io/.../...>]
). Any thoughts on what's causing this?
hi @Alessandro De Rose ! I think if you are loading GCP service credentials into your GitHub action workflow, prefect will infer those credentials at runtime
we use this same pattern internally but use workload identity instead. are you familiar with that? I can share a code snippet shortly!
Hi @Jamie Zieziula , thanks for following up! Yes, I was initially attempting to authenticate to GCloud in the Github action using Workload Identity. But on deploy, Prefect doesn't seems to be attempting to push to
instead of
. Please see my action steps below:
Copy code
      - name: Checkout repository
        uses: actions/checkout@v3

      - id: auth
        name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v1
          workload_identity_provider: ...
          service_account: ...

      - name: Configure Google Cloud credential helper
        run: gcloud auth configure-docker us-docker.pkg.dev

      - uses: actions/setup-python@v4
          python-version: '3.11'
          cache: 'pip'

      - name: Install Prefect
        run: pip install prefect==2.10.9

      - name: Authenticate with Prefect
        run: prefect cloud login --key ${{ secrets.PREFECT_API_KEY }} --workspace ${{ secrets.PREFECT_WORKSPACE }}

      - name: Deploy all flows listed in deployment.yaml
        run: prefect deploy --all
can you share your deployment.yaml file as well?
got it — you can remove everything outside of the deployments: list now btw
last thing - can you share your prefect.yaml
Copy code
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: prefect_flows
prefect-version: 2.10.9

# build section allows you to manage and build docker images
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.2.0
    image_name: ###/###
    tag: latest
    dockerfile: auto
    push: true

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
- prefect.projects.steps.git_clone_project:
    repository: <https://github.com/###/###.git>
    branch: main
    access_token: null
And to confirm your:
Copy code
- prefect_docker.projects.steps.build_docker_image:
    image_name: ###/###
starts with
ah, nope it does not. Let me do that now
cool - that should fix your issue!
Thank you! While I have your attention -- I generated the deployment using
prefect deployment build
-- this currently creates a new file for each deployment. Is there a flag I can use to get it to append to an existing deployment.yaml file?
ah i see - with the introduction of
you would run
prefect project init
when you initially create your project, and then add / update your deployment specs manually rather than via the cli
Understood -- I went for the
approach because I couldn't find much documentation on the required properties. Would you be able to point me in the right direction? The auto-generated deployment.yaml is rather verbose so if I can get rid of some unnecessary fields that would be great (especially in the credentials/storage sections)
Yes definitely - this doc should provide some clarity on different property types for a deployment
Thanks! I can't see any reference to storage/infrastructure there, where would those be defined?
Right right - are you using an agent or worker?
ah thats my bad, I assumed you were using a worker! Ignore me then 🙂 your original deployment spec was right
Hmm that deployment spec didn't seem to be working -- it gives me the following error within the Prefect Cloud UI:
This deployment doesn't have an associated work queue; runs will be scheduled but won't be picked up by your agents
But the queue is defined in the `deployment.yaml`:
Copy code
work_queue_name: default
  work_pool_name: cloud-run-work-pool
Also within the UI, I can see "Storage Document ID" and "Infrastructure Document ID" are
. So it doesn't seem to be picking up those from the
either -- despite it being generated by
prefect deployment build
Yep that makes sense - Projects only support using a
rather than an agent
the deployment of a worker is very similar to an agent - especially if you are using the helm chart for deployment (assuming you’re using a k8s cluster)
you’ll define the
step in
— this will replace the need for a storage block // document
in your work pool, you’ll define infra defaults, which can be overridden later at the individual deployment level — this replaces the need for an infra block // document
brilliant, thanks for all your support!