Balazs
09/09/2025, 4:41 PMMarvin
09/09/2025, 4:41 PMMarvin
09/09/2025, 4:43 PMprefect deploy) to create/upload an artifact (e.g., zip your project and upload to GCS, build/push a Docker image).
- Pull actions: run at run time (on the worker) to fetch what’s needed to execute (e.g., download that artifact from GCS, or just use the container image).
Why use Prefect’s push/build at all?
- Single source of truth in prefect.yaml: Your deployment “recipe” is fully declarative (what to package, where to put it, and how to fetch it). One command works locally and in CI.
- Reproducible snapshots: Prefect will snapshot your project according to your ignore rules, content-hash it, and store it immutably (e.g., gs://.../myflow/<hash>.zip). This reduces “it works on my machine” drift.
- Version + provenance in the UI: The exact artifact/image that was used for a deployment is recorded with logs from push/pull actions, making rollbacks and debugging easier.
- Seamless templating/secrets: Prefect variables/blocks can parameterize paths, tags, and credentials without custom scripting.
- Lower ceremony for teams without existing CI: If you don’t have a mature build pipeline, prefect deploy can take you from code to a runnable deployment quickly.
When using your own CI/CD may be better
- You already have robust pipelines for building images or packaging/uploading artifacts.
- Your build is complex (multi-stage Docker, private build systems, heavy caching).
- Security constraints (build only in hardened CI, not from dev machines).
- You prefer explicit control over tagging/versioning and promotion.
Patterns that work well without Prefect push
- GCS-backed code:
- Your CI zips the project and uploads it to gs://<bucket>/<prefix>/<version>.zip.
- In prefect.yaml, only define a pull action to download that exact object before runs.
- Omit push entirely. Prefect will just pull and run.
- Container images:
- Your CI builds/pushes your-registry/your-image:tag.
- In the deployment (or work pool job variables), set the image to that tag.
- Omit push/build and often omit pull too — the runtime (K8s/ECS/etc.) pulls the image.
So, do you have to use Prefect’s push/build? No.
- It’s there to make deployment lifecycle repeatable and observable from Prefect, but it’s fully optional. Many teams standardize on:
- CI builds the image (and optionally uploads a code artifact),
- prefect.yaml references the image (and/or defines a pull step for blob storage),
- prefect deploy just registers the deployment and links it to the artifact/image you already produced.
If you want, I can sketch a minimal prefect.yaml for:
- “Only-pull from GCS” (assuming CI uploaded `gs://bucket/prefix/<sha>.zip`)
- “Container-only” (image baked in CI, no pull needed)
Docs
- Prefect 3.x prefect.yaml overview: prefect.yaml (Prefect 3)
- Background on push/pull architecture: Push vs Pull overview
Quick clarification to tailor examples:
- Are you planning to run from container images, blob storage, or Git as the source of truth?
- Do you want your workers to download code at run time, or is all code baked into the image?Balazs
09/10/2025, 9:54 AMMarvin
09/10/2025, 9:55 AMMarvin
09/10/2025, 10:01 AMname: gcs-push-pull-demo
prefect-version: ">=3.0.0"
# Optional: centralize values you’ll reuse
variables:
gcp_project: "my-gcp-project"
gcs_bucket: "my-bucket"
gcs_prefix: "prefect/artifacts/gcs-push-pull-demo" # GCS "folder" prefix
# Make sure this file exists and does not ignore your entire project!
# See sample .prefectignore below.
# (push_to_gcs will use this file by default)
# project:
# ignore_file: ".prefectignore"
deployments:
- name: gcs-demo
entrypoint: "flows/flow.py:my_flow"
work_pool:
name: "default-agent-pool"
# Runs during `prefect deploy`
push:
- prefect_gcp.deployments.steps.push_to_gcs:
bucket: "${{ variables.gcs_bucket }}"
folder: "${{ variables.gcs_prefix }}"
project: "${{ variables.gcp_project }}"
# credentials: # Optional; omit to use ADC on the machine running `prefect deploy`
# service_account_info: ${YOUR_SA_JSON}
# Runs on the worker before each flow run
pull:
- prefect_gcp.deployments.steps.pull_from_gcs:
bucket: "${{ variables.gcs_bucket }}"
folder: "${{ variables.gcs_prefix }}"
project: "${{ variables.gcp_project }}"
# credentials: # Optional; omit to use ADC on the worker
# service_account_info: ${YOUR_SA_JSON}
Sample .prefectignore (put this in your project root)
# Common ignores
__pycache__/
*.pyc
.git/
.prefect/
.env
.venv/
node_modules/
.dist-info/
How this works
- push_to_gcs runs when you execute prefect deploy from the project root. It snapshots your project (respecting .prefectignore) and uploads it to gs://<bucket>/<folder> under a content-addressed path (so look under that prefix for your uploaded zip).
- pull_from_gcs runs on the worker before each flow run and downloads/extracts that snapshot so your entrypoint can be executed.
Requirements to make it work
- Install the GCP integration where you run deploy and where workers run:
- pip install "prefect-gcp>=0.5"
- Authenticate with GCP:
- Either ADC (e.g., gcloud auth application-default login or GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa.json)
- Or pass service account credentials via the credentials field in YAML (a dict shaped like your SA JSON).
Troubleshooting “push does nothing, nothing in GCS”
- Check you’re running from the project root with this prefect.yaml and a flow at the specified entrypoint.
- Confirm the step actually runs: run with verbose logs
- PREFECT_LOGGING_LEVEL=DEBUG prefect deploy
- You should see log lines indicating the push step ran (import path: prefect_gcp.deployments.steps.push_to_gcs).
- Verify you’re looking in the right GCS prefix
- Files are uploaded under the folder you set (variables.gcs_prefix above), usually as a zip with a content hash. Check gs://my-bucket/prefect/artifacts/gcs-push-pull-demo/...
- Inspect your .prefectignore
- Overly broad patterns (like “*” or “**/*”) can make the snapshot empty, so nothing gets uploaded.
- Credentials/permissions
- Ensure the deploying machine has write access to the bucket; missing perms sometimes produce minimal logs unless you enable DEBUG.
- Package installed
- If prefect-gcp is missing, the step import would fail. Double-check it’s installed in the environment running prefect deploy.
Docs
- Prefect YAML (3.x): prefect.yaml guide
- GCP integration: prefect-gcp integration
If you paste your current prefect.yaml and the DEBUG output from prefect deploy, I can pinpoint why your push looks successful but nothing shows up in GCS.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by