https://prefect.io
Join Slack
<@ULVA73B9P> - I have a prefect deployment that has suddenly become unable to pull the source code f...
# ask-marvin
j
@Marvin - I have a prefect deployment that has suddenly become unable to pull the source code from a locally-hosted bitbucket I think these sections of prefect.yaml are relevant
Copy code
pull:
- prefect.deployments.steps.pull.pull_with_block:
    id: clone
    block_document_name: "{{ $BITBUCKET_BLOCK_NAME }}"
    block_type_slug: bitbucket-repository
Prefect deployment inspect shows the environment variable is there:
Copy code
'job_variables': {'env': {'BITBUCKET_BLOCK_NAME': 'jds-pipeline-sandbox-bitbucket', 'PREFECT_LOGGING_EXTRA_LOGGERS': 'jds', 'PREFECT_LOGGING_TO_API_WHEN_MISSING_FLOW': 'ignore'}},
and the step seems to be there as well:
Copy code
'pull_steps': [
        {'prefect.deployments.steps.pull.pull_with_block': {'id': 'clone', 'block_type_slug': 'bitbucket-repository', 'block_document_name': '{{ $BITBUCKET_BLOCK_NAME }}'}},
But in the worker logs, the block name doesn't seem to be getting the value injected:
Copy code
ValueError: Unable to find block document named  for block type bitbucket-repository
m
✅ thought for 125.0 seconds
What you’re seeing is consistent with how Prefect 3.x resolves templates in `prefect.yaml`: At deploy time, 
{{ $BITBUCKET_BLOCK_NAME }}
can remain as a template. At runtime on the worker, Prefect tries to resolve it from the worker process environment. If that env var is missing there, Prefect replaces it with an empty string, which leads to:
Copy code
ValueError: Unable to find block document named  for block type bitbucket-repository
So the key point is: 
job_variables.env
is not the same thing as the environment used to render pull steps. Your 
deployment inspect
output shows the deployment has 
BITBUCKET_BLOCK_NAME
in 
job_variables.env
, but the pull step runs before the flow run environment is created, so that variable may not be available when Prefect evaluates:
Copy code
{{ $BITBUCKET_BLOCK_NAME }}
That explains why: - 
deployment inspect
looks correct - worker logs show an empty block name at pull time Most likely root cause The worker itself no longer has 
BITBUCKET_BLOCK_NAME
in its environment, even though the deployment’s infra/job variables do. What to check On the machine/container/pod where the worker is running, verify the worker process environment includes:
Copy code
BITBUCKET_BLOCK_NAME=jds-pipeline-sandbox-bitbucket
If this recently “suddenly” broke, likely causes are: - worker redeployed without that env var - work pool/base job template changed - worker runtime changed (new container image/pod spec/service) - deployment behavior changed because pull steps are being evaluated earlier than the flow-run env injection you expected Recommended fixes Option 1: Put 
BITBUCKET_BLOCK_NAME
in the worker environment This is the most direct fix if you want to keep using 
{{ $BITBUCKET_BLOCK_NAME }}
in 
pull
. Make sure the actual worker process has that env var set, not just the flow run job. Option 2: Hardcode the block name in 
prefect.yaml
 If the block name is static, this is simpler and more reliable:
Copy code
yaml
pull:
  - prefect.deployments.steps.pull.pull_with_block:
      id: clone
      block_document_name: "jds-pipeline-sandbox-bitbucket"
      block_type_slug: bitbucket-repository
Option 3: Use a Prefect Variable instead of an env var If you want it configurable but not dependent on worker env injection:
Copy code
yaml
pull:
  - prefect.deployments.steps.pull.pull_with_block:
      id: clone
      block_document_name: "{{ prefect.variables.bitbucket_block_name }}"
      block_type_slug: bitbucket-repository
That’s often safer for deployment config than 
{{ $ENV_VAR }}
. Why this happens technically Prefect’s runtime step resolution applies environment-variable templating from 
os.environ
on the worker. If the variable is absent, it gets turned into 
""
rather than failing earlier, which is why you end up with:
Copy code
Unable to find block document named  for block type bitbucket-repository
Bottom line Your deployment config is probably fine. The problem is that 
BITBUCKET_BLOCK_NAME
is not available to the worker at pull-step evaluation time. If you want, I can help you pinpoint where to set it correctly based on your worker type: - Docker worker - Kubernetes worker - process worker - ECS/other And I can suggest the exact configuration change.
j
@Marvin This is a recent change in behavior. It definitely used to work. Is there a release note about this somewhere? I am using a locally hosted process worker, and the worker is connnected to prefect cloud
m
✅ thought for 154.2 seconds
I couldn’t find a release note or issue indicating a recent intentional change to this behavior. Based on the current Prefect 3.x code path, the behavior is: - 
pull
steps resolve 
{{ $ENV_VAR }}
from the worker process OS environment - 
job_variables.env
is not used to render pull steps - that means a process worker must have 
BITBUCKET_BLOCK_NAME
set in the environment of the running worker process itself So if this used to work, I’d suspect one of these before a Prefect product change: - the worker used to be started from a shell/session that exported 
BITBUCKET_BLOCK_NAME
- the worker is now started by a service manager (
systemd
, supervisor, launchd, cron, etc.) without that env var - the worker host changed - the user account or startup script changed - the Prefect version changed and exposed a pre-existing configuration gap, but I’m not seeing a release note that says “pull steps now stop seeing job_variables env” Important distinction for a process worker For a locally hosted process worker connected to Prefect Cloud: - 
job_variables.env
applies to the flow run subprocess/job - 
pull
happens before that subprocess environment matters - so 
pull_with_block
only sees env vars from the worker process environment That matches your symptom exactly. What I’d check right now If the worker is running as a long-lived process/service, inspect the environment of that process, not your interactive shell. For example, if started manually:
Copy code
bash
echo $BITBUCKET_BLOCK_NAME
prefect worker start ...
If started via 
systemd
, check the unit file / drop-in for 
Environment=
or 
EnvironmentFile=
. If you want a quick diagnostic, restart the worker from a shell where you explicitly export the variable first:
Copy code
bash
export BITBUCKET_BLOCK_NAME=jds-pipeline-sandbox-bitbucket
prefect worker start --pool <your-pool-name>
If that fixes it, then the issue is definitely worker-process environment, not the deployment itself. I did not find a release note for this Latest notes I checked didn’t mention a change to pull-step env interpolation behavior: Prefect releases You may also want to compare the version you were on before vs. now: Release notes Practical recommendation For a stable fix on a process worker, either: 1. set 
BITBUCKET_BLOCK_NAME
in the worker service environment, or 2. stop templating the block name via env var and use the literal block name directly in 
prefect.yaml
, or 3. use a Prefect Variable for the block name instead of 
{{ $... }}
For your case, since the block name looks static, I’d probably use:
Copy code
yaml
pull:
  - prefect.deployments.steps.pull.pull_with_block:
      id: clone
      block_document_name: "jds-pipeline-sandbox-bitbucket"
      block_type_slug: bitbucket-repository
If you want, I can help you verify this specifically for how your process worker is launched — e.g. shell, 
systemd
, Docker, tmux, etc.
j
@Marvin The environment variable was originally suggested as the workaround since my prefect.yaml has two deployments: sandbox and prod, and they have different bitbucket block names, but the prefect.yaml file only has one "pull" section for all deployments in the file. Can you help me create two separate pull steps for the two deployments?
m
✅ thought for 73.2 seconds
Yes — in Prefect 3.x you can define per-deployment 
pull
steps inside each deployment, and those override the top-level 
pull
. So for your case, the clean fix is: - remove the shared top-level 
pull
- add a separate 
pull
section under each deployment - hardcode the correct Bitbucket block name for each one Example
Copy code
yaml
prefect-version: null
name: my-project

build: null
push: null

deployments:
  - name: sandbox
    entrypoint: flows/my_flow.py:my_flow
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: "jds-pipeline-sandbox-bitbucket"
          block_type_slug: bitbucket-repository
    work_pool:
      name: my-process-pool

  - name: prod
    entrypoint: flows/my_flow.py:my_flow
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: "jds-pipeline-prod-bitbucket"
          block_type_slug: bitbucket-repository
    work_pool:
      name: my-process-pool
If both deployments share most settings, you can also use YAML anchors to keep it tidy. Example with shared config
Copy code
yaml
prefect-version: null
name: my-project

definitions:
  base_deployment: &base_deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-process-pool

deployments:
  - <<: *base_deployment
    name: sandbox
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: "jds-pipeline-sandbox-bitbucket"
          block_type_slug: bitbucket-repository

  - <<: *base_deployment
    name: prod
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: "jds-pipeline-prod-bitbucket"
          block_type_slug: bitbucket-repository
Why this is better than the env-var workaround This avoids relying on the worker’s OS environment for pull-step templating, which is exactly the part that’s biting you now. Notes - You can omit the top-level 
pull
entirely if each deployment defines its own. - Per-deployment 
pull
takes precedence over any top-level 
pull
. - Since your block names are known and static per environment, this is probably the most reliable setup. Docs: prefect.yaml deployment configuration If you want, paste your current 
prefect.yaml
and I can rewrite it into the two-deployment form for you.
j
@Marvin there are three pull steps - 1. pull_with_block to get the source code 2. run_shell_script to do some setup checks 3. pip_install_requirements to install requirements Show all three steps in the example, hopefully sharing code for steps 2 and 3
m
✅ thought for 11.9 seconds
Absolutely — you can keep steps 2 and 3 shared, and only vary step 1 per deployment. A clean way to do that in 
prefect.yaml
is with YAML anchors.
Copy code
yaml
prefect-version: null
name: my-project

definitions:
  shared_pull_steps: &shared_pull_steps
    - prefect.deployments.steps.utility.run_shell_script:
        id: setup_checks
        script: |
          echo "Running setup checks..."
          test -f requirements.txt
    - prefect.deployments.steps.pull.pip_install_requirements:
        directory: .
        requirements_file: requirements.txt

  sandbox_source_pull: &sandbox_source_pull
    - prefect.deployments.steps.pull.pull_with_block:
        id: clone
        block_document_name: "jds-pipeline-sandbox-bitbucket"
        block_type_slug: bitbucket-repository

  prod_source_pull: &prod_source_pull
    - prefect.deployments.steps.pull.pull_with_block:
        id: clone
        block_document_name: "jds-pipeline-prod-bitbucket"
        block_type_slug: bitbucket-repository

  base_deployment: &base_deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-process-pool

deployments:
  - <<: *base_deployment
    name: sandbox
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: "jds-pipeline-sandbox-bitbucket"
          block_type_slug: bitbucket-repository
      - prefect.deployments.steps.utility.run_shell_script:
          id: setup_checks
          script: |
            echo "Running setup checks..."
            test -f requirements.txt
      - prefect.deployments.steps.pull.pip_install_requirements:
          directory: .
          requirements_file: requirements.txt

  - <<: *base_deployment
    name: prod
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: "jds-pipeline-prod-bitbucket"
          block_type_slug: bitbucket-repository
      - prefect.deployments.steps.utility.run_shell_script:
          id: setup_checks
          script: |
            echo "Running setup checks..."
            test -f requirements.txt
      - prefect.deployments.steps.pull.pip_install_requirements:
          directory: .
          requirements_file: requirements.txt
Important note about YAML anchors YAML can reuse mappings nicely, but it does not merge lists the way people often want. So while you can define 
shared_pull_steps
, you generally can’t do “deployment-specific first step + splice in shared list” in plain YAML. That means the most reliable approach is usually: - use anchors for shared mappings like 
base_deployment
- duplicate the shared pull steps in each deployment - or move the shared logic into a single shell script and call that from one shared step So in practice, the example above is the safest 
prefect.yaml
shape. Alternative: put steps 2 and 3 behind one shared shell script If you want less duplication, you could make step 2 call a script in your repo that does both checks and install work:
Copy code
yaml
deployments:
  - <<: *base_deployment
    name: sandbox
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: "jds-pipeline-sandbox-bitbucket"
          block_type_slug: bitbucket-repository
      - prefect.deployments.steps.utility.run_shell_script:
          id: setup
          script: |
            ./scripts/prefect_worker_setup.sh

  - <<: *base_deployment
    name: prod
    pull:
      - prefect.deployments.steps.pull.pull_with_block:
          id: clone
          block_document_name: "jds-pipeline-prod-bitbucket"
          block_type_slug: bitbucket-repository
      - prefect.deployments.steps.utility.run_shell_script:
          id: setup
          script: |
            ./scripts/prefect_worker_setup.sh
Then 
scripts/prefect_worker_setup.sh
could contain: ```bash #!/usr/bin/env bash set -euo pipefail echo "Running setup checks..." test -f requirements.txt
pip install -r requirements.txt ``` That reduces duplication, though it replaces 
pip_install_requirements
with plain 
pip install
. If you want to keep the actual Prefect install step then use the first example and duplicate steps 2 and 3 under each deployment. Docs: prefect.yaml deployment configuration If you want, I can also show you a version using YAML anchors a bit more aggressively, or help adapt this to your exact current 
prefect.yaml
.
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by