https://prefect.io
Join Slack
Hi there, I would like to use `prefect deploy --all` to deploy multiple deployments stored in a `dep...
# prefect-cloud
a
Hi there, I would like to use 
prefect deploy --all
to deploy multiple deployments stored in a 
deployment.yaml
file. I have a build step stored in 
prefect.yaml
to generate an image containing flow dependencies where 
push: true
to ensure that the built image is pushed to Google Artifact Registry (from which Cloud Run infra will pull from). The call to 
deploy
is being carried out in a GitHub action, before calling 
prefect deploy --all
, I have run 
gcloud auth configure-docker <http://us-docker.pkg.dev|us-docker.pkg.dev>
to configure 
gcloud
as the credential helper for docker. However, this action is currently failing with the error 
unauthorized: access token has insufficient scopes
because Prefect seems to be attempting to push to 
<http://docker.io|docker.io>
still (
The push refers to repository [<http://docker.io/.../...|docker.io/.../...>]
). Any thoughts on what's causing this?
✅ 1
j
hi @Alessandro De Rose ! I think if you are loading GCP service credentials into your GitHub action workflow, prefect will infer those credentials at runtime
we use this same pattern internally but use workload identity instead. are you familiar with that? I can share a code snippet shortly!
a
Hi @Jamie Zieziula , thanks for following up! Yes, I was initially attempting to authenticate to GCloud in the Github action using Workload Identity. But on deploy, Prefect doesn't seems to be attempting to push to 
<http://docker.io|docker.io>
instead of 
<http://us-docker.pkg.dev|us-docker.pkg.dev>
. Please see my action steps below:
Copy code
steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - id: auth
        name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v1
        with:
          workload_identity_provider: ...
          service_account: ...

      - name: Configure Google Cloud credential helper
        run: gcloud auth configure-docker us-docker.pkg.dev

      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install Prefect
        run: pip install prefect==2.10.9

      - name: Authenticate with Prefect
        run: prefect cloud login --key ${{ secrets.PREFECT_API_KEY }} --workspace ${{ secrets.PREFECT_WORKSPACE }}

      - name: Deploy all flows listed in deployment.yaml
        run: prefect deploy --all
👀 1
j
can you share your deployment.yaml file as well?
got it — you can remove everything outside of the deployments: list now btw
last thing - can you share your prefect.yaml
a
Copy code
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: prefect_flows
prefect-version: 2.10.9

# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.2.0
    image_name: ###/###
    tag: latest
    dockerfile: auto
    push: true

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.git_clone_project:
    repository: <https://github.com/###/###.git>
    branch: main
    access_token: null
j
And to confirm your:
Copy code
build:
- prefect_docker.projects.steps.build_docker_image:
    image_name: ###/###
starts with 
<http://us-docker.pkg.dev/|us-docker.pkg.dev/>
a
ah, nope it does not. Let me do that now
j
cool - that should fix your issue!
a
Thank you! While I have your attention -- I generated the deployment using 
prefect deployment build
-- this currently creates a new file for each deployment. Is there a flag I can use to get it to append to an existing deployment.yaml file?
j
ah i see - with the introduction of 
projects
you would run 
prefect project init
when you initially create your project, and then add / update your deployment specs manually rather than via the cli
a
Understood -- I went for the 
deployment
approach because I couldn't find much documentation on the required properties. Would you be able to point me in the right direction? The auto-generated deployment.yaml is rather verbose so if I can get rid of some unnecessary fields that would be great (especially in the credentials/storage sections)
j
Yes definitely - this doc should provide some clarity on different property types for a deployment
a
Thanks! I can't see any reference to storage/infrastructure there, where would those be defined?
j
Right right - are you using an agent or worker?
a
agent
j
ah thats my bad, I assumed you were using a worker! Ignore me then 🙂 your original deployment spec was right
a
Hmm that deployment spec didn't seem to be working -- it gives me the following error within the Prefect Cloud UI: 
This deployment doesn't have an associated work queue; runs will be scheduled but won't be picked up by your agents
But the queue is defined in the `deployment.yaml`:
Copy code
work_queue_name: default
  work_pool_name: cloud-run-work-pool
Also within the UI, I can see "Storage Document ID" and "Infrastructure Document ID" are 
None
. So it doesn't seem to be picking up those from the 
deployment.yaml
either -- despite it being generated by 
prefect deployment build
.
j
Yep that makes sense - Projects only support using a 
worker
rather than an agent
the deployment of a worker is very similar to an agent - especially if you are using the helm chart for deployment (assuming you’re using a k8s cluster)
you’ll define the 
pull
step in 
prefect.yaml
— this will replace the need for a storage block // document
in your work pool, you’ll define infra defaults, which can be overridden later at the individual deployment level — this replaces the need for an infra block // document
a
brilliant, thanks for all your support!
j
anytime!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by