<@ULVA73B9P> If I want to deploy from blob storage...
# ask-marvin
b
@Marvin If I want to deploy from blob storage (GCS), what is the advantage of defining the push action in prefect.yml as opposed to just pushing my code myself in CI/CD? Kind of the same question applies to container images as well. Can't I just build my own image and only specify a pull action? I am confused why Prefect has these built in facilities and tries to do code push/ build.
m
thought for 136.9 seconds
Great question — and you’re not alone. Short answer: you absolutely can keep building images and uploading code in your own CI/CD and only use Prefect to pull or just run. Prefect’s push/build actions are optional conveniences, not requirements. What push/pull do in Prefect 3.x - Push actions: run at deploy time (when you run
prefect deploy
) to create/upload an artifact (e.g., zip your project and upload to GCS, build/push a Docker image). - Pull actions: run at run time (on the worker) to fetch what’s needed to execute (e.g., download that artifact from GCS, or just use the container image). Why use Prefect’s push/build at all? - Single source of truth in prefect.yaml: Your deployment “recipe” is fully declarative (what to package, where to put it, and how to fetch it). One command works locally and in CI. - Reproducible snapshots: Prefect will snapshot your project according to your ignore rules, content-hash it, and store it immutably (e.g.,
gs://.../myflow/<hash>.zip
). This reduces “it works on my machine” drift. - Version + provenance in the UI: The exact artifact/image that was used for a deployment is recorded with logs from push/pull actions, making rollbacks and debugging easier. - Seamless templating/secrets: Prefect variables/blocks can parameterize paths, tags, and credentials without custom scripting. - Lower ceremony for teams without existing CI: If you don’t have a mature build pipeline,
prefect deploy
can take you from code to a runnable deployment quickly. When using your own CI/CD may be better - You already have robust pipelines for building images or packaging/uploading artifacts. - Your build is complex (multi-stage Docker, private build systems, heavy caching). - Security constraints (build only in hardened CI, not from dev machines). - You prefer explicit control over tagging/versioning and promotion. Patterns that work well without Prefect push - GCS-backed code: - Your CI zips the project and uploads it to
gs://<bucket>/<prefix>/<version>.zip
. - In prefect.yaml, only define a pull action to download that exact object before runs. - Omit push entirely. Prefect will just pull and run. - Container images: - Your CI builds/pushes
your-registry/your-image:tag
. - In the deployment (or work pool job variables), set the
image
to that tag. - Omit push/build and often omit pull too — the runtime (K8s/ECS/etc.) pulls the image. So, do you have to use Prefect’s push/build? No. - It’s there to make deployment lifecycle repeatable and observable from Prefect, but it’s fully optional. Many teams standardize on: - CI builds the image (and optionally uploads a code artifact), - prefect.yaml references the image (and/or defines a pull step for blob storage), -
prefect deploy
just registers the deployment and links it to the artifact/image you already produced. If you want, I can sketch a minimal prefect.yaml for: - “Only-pull from GCS” (assuming CI uploaded `gs://bucket/prefix/&lt;sha&gt;.zip`) - “Container-only” (image baked in CI, no pull needed) Docs - Prefect 3.x prefect.yaml overview: prefect.yaml (Prefect 3) - Background on push/pull architecture: Push vs Pull overview Quick clarification to tailor examples: - Are you planning to run from container images, blob storage, or Git as the source of truth? - Do you want your workers to download code at run time, or is all code baked into the image?
b
@Marvin can you give me an example prefect.yml for pushing the project to GCS and pulling it as well. I have written a manifest to just use Prefect's built-in push mechanism. Nothing is happening. The push step does not fail, but there is nothing in GCS
m
thought for 381.6 seconds
Got it — here’s a minimal, working Prefect 3.x prefect.yaml that pushes your project to GCS at deploy time and pulls it at run time. Example prefect.yaml
Copy code
name: gcs-push-pull-demo
prefect-version: ">=3.0.0"

# Optional: centralize values you’ll reuse
variables:
  gcp_project: "my-gcp-project"
  gcs_bucket: "my-bucket"
  gcs_prefix: "prefect/artifacts/gcs-push-pull-demo"  # GCS "folder" prefix

# Make sure this file exists and does not ignore your entire project!
# See sample .prefectignore below.
# (push_to_gcs will use this file by default)
# project:
#   ignore_file: ".prefectignore"

deployments:
- name: gcs-demo
  entrypoint: "flows/flow.py:my_flow"
  work_pool:
    name: "default-agent-pool"

  # Runs during `prefect deploy`
  push:
  - prefect_gcp.deployments.steps.push_to_gcs:
      bucket: "${{ variables.gcs_bucket }}"
      folder: "${{ variables.gcs_prefix }}"
      project: "${{ variables.gcp_project }}"
      # credentials:  # Optional; omit to use ADC on the machine running `prefect deploy`
      #   service_account_info: ${YOUR_SA_JSON}

  # Runs on the worker before each flow run
  pull:
  - prefect_gcp.deployments.steps.pull_from_gcs:
      bucket: "${{ variables.gcs_bucket }}"
      folder: "${{ variables.gcs_prefix }}"
      project: "${{ variables.gcp_project }}"
      # credentials:  # Optional; omit to use ADC on the worker
      #   service_account_info: ${YOUR_SA_JSON}
Sample .prefectignore (put this in your project root)
Copy code
# Common ignores
__pycache__/
*.pyc
.git/
.prefect/
.env
.venv/
node_modules/
.dist-info/
How this works - push_to_gcs runs when you execute
prefect deploy
from the project root. It snapshots your project (respecting .prefectignore) and uploads it to gs://<bucket>/<folder> under a content-addressed path (so look under that prefix for your uploaded zip). - pull_from_gcs runs on the worker before each flow run and downloads/extracts that snapshot so your entrypoint can be executed. Requirements to make it work - Install the GCP integration where you run deploy and where workers run: - pip install "prefect-gcp>=0.5" - Authenticate with GCP: - Either ADC (e.g., gcloud auth application-default login or GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa.json) - Or pass service account credentials via the credentials field in YAML (a dict shaped like your SA JSON). Troubleshooting “push does nothing, nothing in GCS” - Check you’re running from the project root with this prefect.yaml and a flow at the specified entrypoint. - Confirm the step actually runs: run with verbose logs -
PREFECT_LOGGING_LEVEL=DEBUG prefect deploy
- You should see log lines indicating the push step ran (import path: prefect_gcp.deployments.steps.push_to_gcs). - Verify you’re looking in the right GCS prefix - Files are uploaded under the folder you set (variables.gcs_prefix above), usually as a zip with a content hash. Check gs://my-bucket/prefect/artifacts/gcs-push-pull-demo/... - Inspect your .prefectignore - Overly broad patterns (like “*” or “**/*”) can make the snapshot empty, so nothing gets uploaded. - Credentials/permissions - Ensure the deploying machine has write access to the bucket; missing perms sometimes produce minimal logs unless you enable DEBUG. - Package installed - If prefect-gcp is missing, the step import would fail. Double-check it’s installed in the environment running
prefect deploy
. Docs - Prefect YAML (3.x): prefect.yaml guide - GCP integration: prefect-gcp integration If you paste your current prefect.yaml and the DEBUG output from
prefect deploy
, I can pinpoint why your push looks successful but nothing shows up in GCS.