<@ULVA73B9P> In my pull step, I am using ```- pref...
# ask-marvin
j
@Marvin In my pull step, I am using
Copy code
- prefect.deployments.steps.pull.pull_with_block:
    id: clone
    block_document_name: my-bitbucket-block
    block_type_slug: bitbucket-repository
Can I override the reference name in my-bitbucket-block in the prefect.yaml file or only in the block itself?
m
thought for 395.7 seconds
Short answer: only in the block. In Prefect 3.x, the
pull_with_block
step only takes
block_document_name
and
block_type_slug
. It does not accept extra kwargs (like
reference
,
branch
, or
tag
) to override the repository block’s reference at deploy time. The branch/tag (“reference”) used for the pull is the value saved on the
BitBucketRepository
block. Options if you need to vary the branch/tag per deployment: - Create separate Bitbucket Repository blocks for each branch/tag you need, and point the deployment at the appropriate block. - Switch to the
git_clone
pull step, which supports overriding the branch or commit in
prefect.yaml
. Example using git_clone:
Copy code
pull:
  - prefect.deployments.steps.pull.git_clone:
      repository: <https://bitbucket.org/your-org/your-repo.git>
      branch: feature/my-branch
      # or pin a commit:
      # commit_sha: abcdef1234567890
      # auth options as needed, e.g.:
      # access_token: ${BITBUCKET_APP_PASSWORD}
Refs: - Pull steps API (Prefect 3.x) - prefect-bitbucket integration - prefect.yaml reference If you share your current
prefect.yaml
, I can show the minimal change to switch that specific step to
git_clone
while keeping your credentials setup.
j
@Marvin OK, I think two blocks is the way to go here. If I have a "prod" deployment on the main branch and a "sandbox" deployment that may be on a different branch, can I have two different pull_with_block steps? The rest of the pull steps are the same in both sandbox and prod. It would be nice to have one prefect.yaml file instead of two, but I can split them.
m
thought for 248.1 seconds
Yes—you can keep a single prefect.yaml and define two deployments with different pull_with_block steps. In Prefect 3.x, a deployment’s pull section replaces the global pull section, so you’ll define the full pull list per deployment. To avoid duplicating the “shared” steps, use YAML anchors for the common steps and reference them in each deployment. Example:
Copy code
# Anything under "definitions" is just YAML; Prefect ignores it but you can store anchors here
definitions:
  steps:
    set_wd: &set_wd
      prefect.deployments.steps.pull.set_working_directory:
        directory: .
    # add more shared steps as needed, e.g. sparse checkout, etc.
    # my_shared_step: &my_shared_step
    #   ...

deployments:
  - name: prod
    entrypoint: flows/my_flow.py:flow
    work_pool:
      name: your-pool
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: my-bitbucket-block-main   # main branch block
          block_type_slug: bitbucket-repository
      - *set_wd
      # - *my_shared_step

  - name: sandbox
    entrypoint: flows/my_flow.py:flow
    work_pool:
      name: your-pool
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: my-bitbucket-block-sandbox  # alternate branch block
          block_type_slug: bitbucket-repository
      - *set_wd
      # - *my_shared_step
Notes: - Each deployment’s pull list is independent; there’s no built-in merge with a global pull. Anchors let you reuse shared steps without duplicating YAML. - You can deploy both or just one from the same file (e.g., filter with
--name
when running
prefect deploy
). Docs: - prefect.yaml guide - Pull steps API If you share your current shared steps, I can drop them into the anchors and produce a ready-to-use prefect.yaml for prod/sandbox.
j
@Marvin do you mean I should move the pull step into the
deployments
section rather than at the top of the
prefect.yaml
file General outline of my prefect.yaml file is
Copy code
name: my-repo-deployments
prefect-version: 3.5 

build: null
push: null
pull:
- prefect.deployments.steps.pull.pull_with_block
   ...
- prefect.deployments.steps.utility.run_shell_script
   ...
- prefect.deployment.steps.utility.pip_install_requirements
   ...

deployments:
- name: sandbox
  entrypoint: path/to/somewhere.py:flow_name
  parameters: {
      "param1": "sandbox_value"
  }
  work_pool: sandbox-workers
  work_queue_name: sandbox-queue

- name: prod
  entrypoint: path/to/somewhere.py:flow_name
  parameters: {
     "param1": "prod_value"
  }
  work_pool: prod-workers
  work_queue_name: prod-queue
m
thought for 60.9 seconds
Short answer: yes. If a deployment defines its own
pull
section, it replaces the root-level
pull
for that deployment. So for prod/sandbox with different
pull_with_block
values, put a
pull
list under each deployment. To avoid duplicating the shared steps, use YAML anchors for the common steps and reference them in each deployment’s
pull
list. Example adapted to your outline (note the module path is
prefect.deployments.steps.utility
, plural):
Copy code
name: my-repo-deployments
prefect-version: 3.5

build: null
push: null

# Optional: keep a root-level pull as the default for deployments that don't override it
# pull:
#   - prefect.deployments.steps.pull.pull_with_block:
#       block_document_name: my-bitbucket-block-main
#       block_type_slug: bitbucket-repository
#   - prefect.deployments.steps.utility.run_shell_script:
#       script: echo "do something shared"
#   - prefect.deployments.steps.utility.pip_install_requirements:
#       requirements_file: requirements.txt

# Use a definitions section to hold YAML anchors for shared steps
definitions:
  shared_steps:
    run_script: &run_script
      prefect.deployments.steps.utility.run_shell_script:
        script: echo "do something shared"
    install_reqs: &install_reqs
      prefect.deployments.steps.utility.pip_install_requirements:
        requirements_file: requirements.txt

deployments:
  - name: sandbox
    entrypoint: path/to/somewhere.py:flow_name
    parameters:
      param1: sandbox_value
    work_pool: sandbox-workers
    work_queue_name: sandbox-queue
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          block_document_name: my-bitbucket-block-sandbox
          block_type_slug: bitbucket-repository
      - *run_script
      - *install_reqs

  - name: prod
    entrypoint: path/to/somewhere.py:flow_name
    parameters:
      param1: prod_value
    work_pool: prod-workers
    work_queue_name: prod-queue
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          block_document_name: my-bitbucket-block-main
          block_type_slug: bitbucket-repository
      - *run_script
      - *install_reqs
Tips: - You can keep a root-level
pull
for the “default” branch/block and only add a per-deployment
pull
for the one that differs. Any deployment without a
pull
inherits the root-level one. - When you do define a per-deployment
pull
, list all steps you need—there’s no merging with the root-level list. Refs: - prefect.yaml guide - Pull steps API If you paste your exact shared steps, I’ll drop them into anchors and return a ready-to-use file.
j
@Marvin Can only steps be shared , or can I share the env for the job_variables?
m
thought for 259.6 seconds
You can share more than just steps—you can absolutely share job_variables (including env) across deployments in a single prefect.yaml by using YAML anchors. There isn’t a built-in “global job_variables” section, but anchors let you define once and reuse/override per deployment. Pattern: - Define reusable job_variables/env in a definitions section as anchors - Merge them into each deployment’s work_pool.job_variables - Override just the bits that differ (e.g., one or two env vars) Example:
Copy code
name: my-repo-deployments
prefect-version: 3.5

definitions:
  job_vars:
    base_env: &base_env
      COMMON_FLAG: "1"
      API_TOKEN: "{{ $API_TOKEN }}"  # pulled from your environment at deploy time
    docker_job: &docker_job
      image: my-org/my-image:latest
      env: *base_env

deployments:
  - name: sandbox
    entrypoint: path/to/somewhere.py:flow_name
    parameters:
      param1: sandbox_value
    work_pool:
      name: sandbox-workers
      job_variables:
        <<: *docker_job
        env:
          <<: *base_env
          APP_MODE: sandbox
    work_queue_name: sandbox-queue

  - name: prod
    entrypoint: path/to/somewhere.py:flow_name
    parameters:
      param1: prod_value
    work_pool:
      name: prod-workers
      job_variables:
        <<: *docker_job
        env:
          <<: *base_env
          APP_MODE: prod
    work_queue_name: prod-queue
Notes: - Deployment-level job_variables override the work pool’s base job template. The anchor/merge here is just YAML reuse to keep your file DRY. - The exact shape of job_variables (and env) is worker-type specific; the above is a common pattern for Docker or Process workers. - You can still keep a root-level pull for defaults and define per-deployment pull only when it differs. Docs: - prefect.yaml guide - Customize job variables If you tell me which worker type you’re using (Process, Docker, Kubernetes), I can tailor the env structure and a ready-to-paste snippet.
j
@Marvin I am trying to have separate steps for each deployment in my prefect.yaml and I'm getting the error: Invalid fields in deployments: • myapp-prod: pull • myapp-sandbox: pull
m
thought for 394.8 seconds
Thanks for the details — that error makes sense with Prefect 3.5. What’s happening - In Prefect 3.5,
pull
is not a valid key inside an individual deployment definition. Only the top-level
pull
section is supported. That’s why you’re seeing: • myapp-prod: pull • myapp-sandbox: pull How to keep one prefect.yaml and vary just the repo/block per deployment Since your only difference is which Bitbucket block to use (and the rest of the pull steps are identical), keep a single top-level
pull
and parameterize the block name via an environment variable. Then set that env var differently per deployment using
work_pool.job_variables.env
. Example:
Copy code
name: my-repo-deployments
prefect-version: 3.5

# One global pull section used by all deployments
pull:
  - prefect.deployments.steps.pull.pull_with_block:
      id: clone
      block_document_name: "{{ $BITBUCKET_BLOCK_NAME }}"
      block_type_slug: bitbucket-repository
  - prefect.deployments.steps.utility.run_shell_script:
      script: echo "shared step runs for both"
  - prefect.deployments.steps.utility.pip_install_requirements:
      requirements_file: requirements.txt

deployments:
  - name: myapp-sandbox
    entrypoint: path/to/somewhere.py:flow_name
    parameters:
      param1: sandbox_value
    work_pool:
      name: sandbox-workers
      job_variables:
        env:
          BITBUCKET_BLOCK_NAME: my-bitbucket-block-sandbox
    work_queue_name: sandbox-queue

  - name: myapp-prod
    entrypoint: path/to/somewhere.py:flow_name
    parameters:
      param1: prod_value
    work_pool:
      name: prod-workers
      job_variables:
        env:
          BITBUCKET_BLOCK_NAME: my-bitbucket-block-main
    work_queue_name: prod-queue
Notes - The
{{ $VAR }}
syntax reads environment variables at runtime for pull steps (the worker’s environment). Using
job_variables.env
ensures the worker gets the correct value per deployment. - If you truly need different step lists per deployment (not just different inputs), with 3.5 you’ll need separate projects/files. If upgrading to a newer 3.x release is an option, I can check whether per-deployment build/push/pull overrides are supported in that version. Docs - prefect.yaml guide - Pull steps API If you share your exact block names and any other shared steps, I can tailor the YAML so you can copy/paste it.