https://prefect.io logo
Powered by
Title
a

Alessandro De Rose

05/15/2023, 5:20 PM
Hi there, I would like to use 
prefect deploy --all
to deploy multiple deployments stored in a 
deployment.yaml
file. I have a build step stored in 
prefect.yaml
to generate an image containing flow dependencies where 
push: true
to ensure that the built image is pushed to Google Artifact Registry (from which Cloud Run infra will pull from). The call to 
deploy
is being carried out in a GitHub action, before calling 
prefect deploy --all
, I have run 
gcloud auth configure-docker <http://us-docker.pkg.dev|us-docker.pkg.dev>
to configure 
gcloud
as the credential helper for docker. However, this action is currently failing with the error 
unauthorized: access token has insufficient scopes
because Prefect seems to be attempting to push to 
<http://docker.io|docker.io>
still (
The push refers to repository [<http://docker.io/.../...|docker.io/.../...>]
). Any thoughts on what's causing this?
j

Jamie Zieziula

05/15/2023, 8:06 PM
hi @Alessandro De Rose ! I think if you are loading GCP service credentials into your GitHub action workflow, prefect will infer those credentials at runtime
we use this same pattern internally but use workload identity instead. are you familiar with that? I can share a code snippet shortly!
a

Alessandro De Rose

05/15/2023, 8:18 PM
Hi @Jamie Zieziula , thanks for following up! Yes, I was initially attempting to authenticate to GCloud in the Github action using Workload Identity. But on deploy, Prefect doesn't seems to be attempting to push to 
<http://docker.io|docker.io>
instead of 
<http://us-docker.pkg.dev|us-docker.pkg.dev>
. Please see my action steps below: 
steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - id: auth
        name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v1
        with:
          workload_identity_provider: ...
          service_account: ...

      - name: Configure Google Cloud credential helper
        run: gcloud auth configure-docker us-docker.pkg.dev

      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install Prefect
        run: pip install prefect==2.10.9

      - name: Authenticate with Prefect
        run: prefect cloud login --key ${{ secrets.PREFECT_API_KEY }} --workspace ${{ secrets.PREFECT_WORKSPACE }}

      - name: Deploy all flows listed in deployment.yaml
        run: prefect deploy --all
j

Jamie Zieziula

05/15/2023, 8:20 PM
can you share your deployment.yaml file as well?
got it — you can remove everything outside of the deployments: list now btw
last thing - can you share your prefect.yaml
a

Alessandro De Rose

05/15/2023, 8:26 PM
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: prefect_flows
prefect-version: 2.10.9

# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.2.0
    image_name: ###/###
    tag: latest
    dockerfile: auto
    push: true

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.git_clone_project:
    repository: <https://github.com/###/###.git>
    branch: main
    access_token: null
j

Jamie Zieziula

05/15/2023, 8:27 PM
And to confirm your: 
build:
- prefect_docker.projects.steps.build_docker_image:
    image_name: ###/###
starts with 
<http://us-docker.pkg.dev/|us-docker.pkg.dev/>
a

Alessandro De Rose

05/15/2023, 8:27 PM
ah, nope it does not. Let me do that now
j

Jamie Zieziula

05/15/2023, 8:29 PM
cool - that should fix your issue!
a

Alessandro De Rose

05/15/2023, 8:33 PM
Thank you! While I have your attention -- I generated the deployment using 
prefect deployment build
-- this currently creates a new file for each deployment. Is there a flag I can use to get it to append to an existing deployment.yaml file?
j

Jamie Zieziula

05/15/2023, 8:39 PM
ah i see - with the introduction of 
projects
you would run 
prefect project init
when you initially create your project, and then add / update your deployment specs manually rather than via the cli
a

Alessandro De Rose

05/15/2023, 8:41 PM
Understood -- I went for the 
deployment
approach because I couldn't find much documentation on the required properties. Would you be able to point me in the right direction? The auto-generated deployment.yaml is rather verbose so if I can get rid of some unnecessary fields that would be great (especially in the credentials/storage sections)
j

Jamie Zieziula

05/15/2023, 8:59 PM
Yes definitely - this doc should provide some clarity on different property types for a deployment
a

Alessandro De Rose

05/15/2023, 9:02 PM
Thanks! I can't see any reference to storage/infrastructure there, where would those be defined?
j

Jamie Zieziula

05/15/2023, 9:05 PM
Right right - are you using an agent or worker?
a

Alessandro De Rose

05/15/2023, 9:06 PM
agent
j

Jamie Zieziula

05/15/2023, 9:07 PM
ah thats my bad, I assumed you were using a worker! Ignore me then 🙂 your original deployment spec was right
a

Alessandro De Rose

05/15/2023, 9:15 PM
Hmm that deployment spec didn't seem to be working -- it gives me the following error within the Prefect Cloud UI: 
This deployment doesn't have an associated work queue; runs will be scheduled but won't be picked up by your agents
But the queue is defined in the `deployment.yaml`: 
work_queue_name: default
  work_pool_name: cloud-run-work-pool
Also within the UI, I can see "Storage Document ID" and "Infrastructure Document ID" are 
None
. So it doesn't seem to be picking up those from the 
deployment.yaml
either -- despite it being generated by 
prefect deployment build
.
j

Jamie Zieziula

05/15/2023, 10:08 PM
Yep that makes sense - Projects only support using a 
worker
rather than an agent
the deployment of a worker is very similar to an agent - especially if you are using the helm chart for deployment (assuming you’re using a k8s cluster)
you’ll define the 
pull
step in 
prefect.yaml
— this will replace the need for a storage block // document
in your work pool, you’ll define infra defaults, which can be overridden later at the individual deployment level — this replaces the need for an infra block // document
a

Alessandro De Rose

05/15/2023, 10:24 PM
brilliant, thanks for all your support!
j

Jamie Zieziula

05/15/2023, 10:54 PM
anytime!
#prefect-cloudJoin Slack
Powered by