<@ULVA73B9P> give me the github actions yaml file ...
# ask-marvinf
Florent VanDeMoortele
12/22/2025, 5:36 PM@Marvin give me the github actions yaml file to deploy my Prefect projet (I build a docker image and upload it to GCS) on Prefect cloud
m
Marvin
12/22/2025, 5:36 PM✅ thought for 130.7 seconds
Marvin
12/22/2025, 5:38 PMGreat — I can share a ready-to-use GitHub Actions workflow for Prefect 2 on Prefect Cloud. Quick clarifier to make sure we nail it:
- Do you truly want to upload the built Docker image as a tar to GCS, or do you actually push images to Artifact Registry/GCR? Prefect workers typically pull images from a container registry. Uploading an image tar to GCS is unusual unless you have a custom loader on the worker side.
Below is an opinionated workflow that:
- Builds a Docker image
- Pushes it to Google Artifact Registry (recommended)
- Optionally also uploads a tarball to GCS (if you need it)
- Updates your Docker infrastructure block in Prefect Cloud to reference the new image
- Deploys using your Prefect 2 “project” (prefect.yaml) via prefect deploy
Assumptions and what you need to provide:
- You are on Prefect 2.x and have a prefect.yaml in your repo defining your deployments.
- You already created a Docker infrastructure block in Prefect Cloud (example name: docker-prod).
- You have a Work Pool configured in your project/deployments (in prefect.yaml).
- GitHub secrets:
- PREFECT_API_KEY
- PREFECT_API_URL (Cloud workspace API URL)
- GCP_CREDENTIALS (JSON key for a service account with writer perms to Artifact Registry and GCS)
- GitHub variables (or replace inline):
- GCP_PROJECT_ID
- GAR_REGION (e.g. us-central1)
- GAR_REPO (Artifact Registry repo name)
- GCS_BUCKET (optional, for the tar upload path)
- PREFECT_DOCKER_BLOCK (name of your Docker infra block, e.g. docker-prod)
- IMAGE_NAME (e.g. my-flow)
Save this as .github/workflows/prefect-deploy.yml:
```
name: Build and Deploy Prefect 2 to Cloud
on:
push:
branches: [ "main" ]
workflow_dispatch:
permissions:
contents: read
id-token: write # required for google-github-actions/auth if using WIF later
env:
# Set via GitHub Repository Variables (preferred) or replace inline
GCP_PROJECT_ID: ${{ vars.GCP_PROJECT_ID }}
GAR_REGION: ${{ vars.GAR_REGION }} # e.g. us-central1
GAR_REPO: ${{ vars.GAR_REPO }} # e.g. my-artifacts
IMAGE_NAME: ${{ vars.IMAGE_NAME }} # e.g. my-flow
GCS_BUCKET: ${{ vars.GCS_BUCKET }} # optional: your GCS bucket name
PREFECT_DOCKER_BLOCK: ${{ vars.PREFECT_DOCKER_BLOCK }} # e.g. docker-prod
# Prefect Cloud auth
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }} # Cloud workspace API URL
jobs:
build-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Prefect 2 and GCP collection
run: |
python -m pip install --upgrade pip
pip install "prefect>=2,<3" prefect-gcp
- name: Authenticate to Google Cloud (service account JSON)
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GCP_CREDENTIALS }}
- name: Set up gcloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
project_id: ${{ env.GCP_PROJECT_ID }}
- name: Compute image tags
run: |
echo "IMAGE_TAG=${GITHUB_SHA::12}" >> $GITHUB_ENV
echo "REGISTRY_HOST=${{ env.GAR_REGION }}-docker.pkg.dev" >> $GITHUB_ENV
echo "FULL_IMAGE=${{ env.GAR_REGION }}-docker.pkg.dev/${{ env.GCP_PROJECT_ID }}/${{ env.GAR_REPO }}/${{ env.IMAGE_NAME }}:${GITHUB_SHA::12}" >> $GITHUB_ENV
echo "FULL_IMAGE_LATEST=${{ env.GAR_REGION }}-docker.pkg.dev/${{ env.GCP_PROJECT_ID }}/${{ env.GAR_REPO }}/${{ env.IMAGE_NAME }}:latest" >> $GITHUB_ENV
- name: Configure Docker auth for Artifact Registry
run: |
gcloud auth configure-docker $REGISTRY_HOST --quiet
- name: Build image
run: |
docker build -t "$FULL_IMAGE" -t "$FULL_IMAGE_LATEST" .
- name: Push image to Artifact Registry
run: |
Marvin
12/22/2025, 5:38 PMdocker push "$FULL_IMAGE"
docker push "$FULL_IMAGE_LATEST"
# Optional: also upload a compressed image tar to GCS (only if you actually need it)
- name: Optionally upload image tar to GCS
if: env.GCS_BUCKET != ''
run: |
docker save "$FULL_IMAGE" | gzip > image.tar.gz
gsutil cp image.tar.gz gs://$GCS_BUCKET/prefect-images/${{ env.IMAGE_NAME }}-${{ env.IMAGE_TAG }}.tar.gz
- name: Verify Prefect auth
run: prefect version
- name: Update Docker infrastructure block image in Prefect Cloud
env:
FULL_IMAGE: ${{ env.FULL_IMAGE }}
PREFECT_DOCKER_BLOCK: ${{ env.PREFECT_DOCKER_BLOCK }}
run: |
python - <<'PY'
import os
from prefect.infrastructure import DockerContainer
block_name = os.environ.get("PREFECT_DOCKER_BLOCK", "docker-prod")
image = os.environ["FULL_IMAGE"]
block = DockerContainer.load(block_name)
block.image = image
block.save(block_name, overwrite=True)
print(f"Updated Docker infrastructure block '{block_name}' to image: {image}")
PY
- name: Deploy all deployments from prefect.yaml
run: |
# Assumes your repository has a prefect.yaml describing your deployments
# and that those deployments reference the Docker block updated above.
prefect deploy --all
```
How to wire this up:
- Create a Docker infrastructure block in Prefect Cloud (e.g., docker-prod) and reference that block in your prefect.yaml deployments.
- Store PREFECT_API_KEY and PREFECT_API_URL as GitHub secrets. In Prefect Cloud, you can copy your Workspace API URL from your workspace settings.
- Store GCP_CREDENTIALS as a GitHub secret containing the service account JSON with:
- roles/artifactregistry.writer (for the target repo)
- roles/storage.objectAdmin (if using the optional GCS tar upload)
- Add GitHub variables for GCP_PROJECT_ID, GAR_REGION, GAR_REPO, IMAGE_NAME, GCS_BUCKET (optional), PREFECT_DOCKER_BLOCK.
Notes:
- The image is tagged with both the commit SHA (immutable) and latest. The block gets updated to the SHA tag to ensure deterministic runs; if you prefer to always run latest, change FULL_IMAGE to use the latest tag when updating the block.
- If you truly need to run from an image tar in GCS, you must have a custom runtime on your workers that pulls and docker load’s that tar before execution. Otherwise, prefer pushing to Artifact Registry and letting workers pull by name.
Helpful references:
- Google Auth GitHub Action: google-github-actions/auth
- Prefect GCP collection: prefect-gcp collection
- Example on defining deployments via prefect.yaml (Prefect 2 Projects): Deployments in Prefect 2
If you share your:
- Artifact Registry path (region/repo),
- GCS bucket name (if you need the tar upload),
- Docker infra block name,
- The file:flow you want to deploy (or confirm you’re using prefect.yaml),
…I can tailor this to your repo and flow entrypoints.
f
Florent VanDeMoortele
12/22/2025, 5:39 PM@Marvin and for Prefect 3?
m
Marvin
12/22/2025, 5:40 PM✅ thought for 178.2 seconds
Marvin
12/22/2025, 5:42 PMAbsolutely — here’s a GitHub Actions workflow tailored for Prefect 3.x on Prefect Cloud that:
- Builds a Docker image
- Pushes it to Google Artifact Registry (recommended)
- Optionally uploads a tarball to GCS
- Deploys your project with
prefect deploy Copy code
name: Build and Deploy Prefect 3 to Cloud
on:
push:
branches: [ "main" ]
workflow_dispatch:
permissions:
contents: read
id-token: write
env:
# Set via GitHub Repository Variables (preferred) or replace inline
GCP_PROJECT_ID: ${{ vars.GCP_PROJECT_ID }}
GAR_REGION: ${{ vars.GAR_REGION }} # e.g. us-central1
GAR_REPO: ${{ vars.GAR_REPO }} # e.g. flows
IMAGE_NAME: ${{ vars.IMAGE_NAME }} # e.g. my-flow
WORK_POOL_NAME: ${{ vars.WORK_POOL_NAME }} # e.g. docker-pool or kubernetes-pool
GCS_BUCKET: ${{ vars.GCS_BUCKET }} # optional: bucket to also upload image tar
# Prefect Cloud auth
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }} # Workspace API URL
jobs:
build-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Prefect 3
run: |
python -m pip install --upgrade pip
pip install "prefect>=3,<4"
- name: Authenticate to Google Cloud (service account JSON)
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GCP_CREDENTIALS }}
- name: Set up gcloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
project_id: ${{ env.GCP_PROJECT_ID }}
- name: Compute image tags
run: |
echo "IMAGE_TAG=${GITHUB_SHA::12}" >> $GITHUB_ENV
echo "REGISTRY_HOST=${{ env.GAR_REGION }}-docker.pkg.dev" >> $GITHUB_ENV
echo "FULL_IMAGE=${{ env.GAR_REGION }}-docker.pkg.dev/${{ env.GCP_PROJECT_ID }}/${{ env.GAR_REPO }}/${{ env.IMAGE_NAME }}:${GITHUB_SHA::12}" >> $GITHUB_ENV
echo "FULL_IMAGE_LATEST=${{ env.GAR_REGION }}-docker.pkg.dev/${{ env.GCP_PROJECT_ID }}/${{ env.GAR_REPO }}/${{ env.IMAGE_NAME }}:latest" >> $GITHUB_ENV
- name: Configure Docker auth for Artifact Registry
run: |
gcloud auth configure-docker $REGISTRY_HOST --quiet
- name: Build image
run: |
docker build -t "$FULL_IMAGE" -t "$FULL_IMAGE_LATEST" .
- name: Push image to Artifact Registry
run: |
docker push "$FULL_IMAGE"
docker push "$FULL_IMAGE_LATEST"
# Optional: also upload a compressed image tar to GCS (only if you truly need it)
- name: Optionally upload image tar to GCS
if: env.GCS_BUCKET != ''
run: |
docker save "$FULL_IMAGE" | gzip > image.tar.gz
gsutil cp image.tar.gz gs://$GCS_BUCKET/prefect-images/${{ env.IMAGE_NAME }}-${{ env.IMAGE_TAG }}.tar.gz
- name: Verify Prefect auth
run: prefect version
# Deploy using prefect.yaml in the repo; override the image via job variables
# This works for Docker or Kubernetes work pools — both accept an "image" job variable.
- name: Deploy from prefect.yaml with image override
run: |
# Deploy all defined deployments in prefect.yaml, setting work pool and image
prefect deploy --all \
--pool "${WORK_POOL_NAME}" \
-jv "image=${FULL_IMAGE}"-jv image=...-jv 'namespace=prefect'-jv 'cpu_request=500m' -jv 'memory_request=512Mi'Marvin
12/22/2025, 5:42 PM-
-jv 'env={"MY_VAR":"value"}'PREFECT_API_KEYPREFECT_API_URLGCP_CREDENTIALSGCP_PROJECT_IDGAR_REGIONGAR_REPOIMAGE_NAMEWORK_POOL_NAMEGCS_BUCKETprefect.yamlf
Florent VanDeMoortele
12/22/2025, 5:44 PM@Marvin here is my github actions that worked for Prefect 2. How can I fix it to work with Prefect 3?
Copy code
name: Deployment
run-name: Deployment for ${{ github.repository }} from ${{ github.ref_name }} by @${{ github.actor }} on ${{ github.event.inputs.environment == 'prod' && 'prod' || 'dev' }} environment
on:
release:
types: [ published ]
workflow_dispatch:
inputs:
environment:
description: The environment to deploy to
required: true
type: choice
options:
- dev
- prod
default: dev
jobs:
tests:
uses: ./.github/workflows/unit_tests.yaml
secrets: inherit
deploy:
needs: [tests]
runs-on: ubuntu-latest
environment: ${{ github.event.inputs.environment == 'prod' && 'prod' || 'dev' }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
id: setup-python
uses: actions/setup-python@v5
with:
python-version: 3.14
- name: Load cached Poetry installation
id: cached-poetry
uses: actions/cache@v4
with:
path: ~/.local
key: poetry-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}
- name: Install Poetry
if: steps.cached-poetry.outputs.cache-hit != 'true'
uses: snok/install-poetry@v1
with:
version: 2.2.1
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true
- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
- name: Install dependencies
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
run: poetry install --no-interaction
- name: Prefect Auth
uses: PrefectHQ/actions-prefect-auth@v1
with:
prefect-api-key: ${{ secrets.PREFECT_API_KEY }}
prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }}
- name: GCP authentication
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GCP_CREDENTIALS }}
- name: Configure Google Cloud credential helper
run: gcloud auth configure-docker --quiet ${{ secrets.GCP_REGION }}-docker.pkg.dev
- name: Set Iguazu variables
run: |
echo "PROJECT_NAME=$(poetry version | awk '{print $1}')" >> $GITHUB_ENV
echo "PROJECT_VERSION=$(poetry version | awk '{print $2}')" >> $GITHUB_ENV
echo "PREFECT_VERSION=$(poetry show | grep "prefect " | awk '{print $2}')" >> $GITHUB_ENV
echo "PREFECT_ACCOUNT_ID=$(prefect config view | tr -d "\n" | awk -F"PREFECT_API_URL='" '{print $2}' | awk -F"'" '{print $1}' | awk -F"/" '{print $6}')" >> $GITHUB_ENV
echo "PREFECT_WORKSPACE_ID=$(prefect config view | tr -d "\n" | awk -F"PREFECT_API_URL='" '{print $2}' | awk -F"'" '{print $1}' | awk -F"/" '{print $8}')" >> $GITHUB_ENV
- name: Set secrets variables
run: |
echo "DEPLOYMENT_ENV=${{ github.event.inputs.environment == 'prod' && 'prod' || 'dev' }}" >> $GITHUB_ENV
echo "GCP_PROJECT_ID=${{ secrets.GCP_PROJECT_ID }}" >> $GITHUB_ENV
echo "GCP_REGION=${{ secrets.GCP_REGION }}" >> $GITHUB_ENV
- name: Prefect Deploy Blocks
run: poetry run prefect block register -m iguazu.blocks.papi
- name: Prefect Deploy Iguazu project
run: poetry run prefect --no-prompt deploy --allm
Marvin
12/22/2025, 5:45 PM✅ thought for 91.8 seconds
Marvin
12/22/2025, 5:46 PMGreat question — you’re close! Here’s how to adapt your Prefect 2 pipeline to Prefect 3 with minimum change.
Key changes from Prefect 2 → 3:
- Keep using
prefect deployprefect deployment buildimageiguazu.blocks.papiprefect.yamlprefect.yamlMarvin
12/22/2025, 5:46 PMecho "PREFECT_WORKSPACE_ID=$(prefect config view | tr -d "\n" | awk -F"PREFECT_API_URL='" '{print $2}' | awk -F"'" '{print $1}' | awk -F"/" '{print $8}')" >> $GITHUB_ENV
- name: Set secrets variables
run: |
echo "DEPLOYMENT_ENV=${{ github.event.inputs.environment == 'prod' && 'prod' || 'dev' }}" >> $GITHUB_ENV
echo "GCP_PROJECT_ID=${{ secrets.GCP_PROJECT_ID }}" >> $GITHUB_ENV
echo "GCP_REGION=${{ secrets.GCP_REGION }}" >> $GITHUB_ENV
# Keep this ONLY if iguazu.blocks.papi contains non-infrastructure blocks you still use (e.g., creds)
- name: Prefect Register Custom Blocks
run: poetry run prefect block register -m iguazu.blocks.papi
- name: Verify Prefect
run: poetry run prefect version
- name: Prefect Deploy (Prefect 3)
run: poetry run prefect --no-prompt deploy --all
# This relies on your updated Prefect 3 prefect.yaml to define build/push steps and work pool/job variables
Copy code
Update your prefect.yaml for Prefect 3
Your workflow relies on prefect.yaml to build/push images and define the work pool. In Prefect 3, use steps like these: Copy code
If you want to override the image from the Action instead of relying on prefect.yaml, change the deploy step to:imagework_pool.job_variables.imageprefect deployment buildDeployment.build_from_flow()f
Florent VanDeMoortele
12/22/2025, 5:52 PM@Marvin it's not true : Prefect support Python 3.14 now
m
Marvin
12/22/2025, 5:52 PM✅ thought for 162.9 seconds
Marvin
12/22/2025, 5:54 PMYou’re right — Prefect 3 supports Python 3.14. Thanks for the correction!
Here’s your workflow adapted for Prefect 3 while keeping Python 3.14 and your Poetry setup. The key changes to ensure Prefect 3 compatibility are:
- Keep using
prefect deployprefect = ">=3,<4"prefect.yamlMarvin
12/22/2025, 5:54 PM# Keep this only if iguazu.blocks.papi defines non-infrastructure blocks you still use (e.g., creds)
- name: Prefect Register Custom Blocks
run: poetry run prefect block register -m iguazu.blocks.papi
- name: Verify Prefect
run: poetry run prefect version
- name: Prefect Deploy (Prefect 3)
run: poetry run prefect --no-prompt deploy --all
# This expects your Prefect 3 prefect.yaml to define build/push steps and work_pool.job_variables.image
Copy code
Double-check your prefect.yaml (Prefect 3 format) includes:
- Docker build/push steps (if you build/push in deploy)
- A work pool, and job variables set for the image used by your workers, e.g.: Copy code
If you prefer to override the image from the Action instead of prefect.yaml, you can:prefect.yaml17 Views