<@ULVA73B9P> I want to be able to run a local pref...
# ask-marvin
a
@Marvin I want to be able to run a local prefect/worker infrastructure for development (using a "local" pull) but then also deploy to prefect cloud with a different runner, such as docker. In this case my deployments will be the same, but the source will be different. what's the recommended way to handle these two separate configurations? Two files with everything duplicated except the pull section? yaml templating? something else?
m
thought for 415.6 seconds
Great question. In Prefect 3.x the most ergonomic way to do this is a single prefect.yaml with two deployments that share all common config and only override what differs (pull/source and work pool). Use YAML anchors to avoid duplication. Here’s a minimal pattern you can copy:
Copy code
# prefect.yaml

definitions:
  shared: &shared
    entrypoint: flows/my_flow.py:my_flow
    parameters:
      log_level: INFO

  local_pool: &local_pool
    name: local

  docker_pool: &docker_pool
    name: docker-prod
    job_variables:
      image: "{{ build-image.image }}"

  local_pull: &local_pull
    - prefect.deployments.steps.set_working_directory:
        directory: .

  git_pull: &git_pull
    - prefect.deployments.steps.git_clone:
        repository: "<https://github.com/your-org/your-repo.git>"
        branch: main

# Global build step (used by the docker deployment)
build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.5.0
      image_name: your-repo/your-image
      tag: "{{ $IMAGE_TAG | default('dev') }}"
      dockerfile: auto

deployments:
  # Local dev: run with local worker, use working dir as source
  - name: my-flow-dev
    <<: *shared
    tags: [dev]
    work_pool: *local_pool
    pull: *local_pull
    build: null   # skip docker build for dev
    push: null    # skip docker push

  # Cloud/prod: docker worker, pull code from git, push image
  - name: my-flow-prod
    <<: *shared
    tags: [prod]
    work_pool: *docker_pool
    pull: *git_pull
    push:
      - prefect_docker.deployments.steps.push_docker_image:
          requires: prefect-docker>=0.5.0
          image_name: "{{ build-image.image_name }}"
          tag: "{{ build-image.tag }}"
Deploy either one without duplicating the whole file:
Copy code
prefect deploy -n my-flow-dev
prefect deploy -n my-flow-prod
Notes and options: - This keeps one source of truth and only varies the pull and work_pool sections. - If you want to avoid hardcoding environment-specific values, parameterize with env vars in templating (e.g.,
{{ $IMAGE_TAG }}
,
{{ $GIT_BRANCH }}
) and set them in CI/CD. - If you truly need two separate files, you can do that too and select with
--prefect-file
, but the single-file + anchors approach usually scales better. - Make sure you have the docker integration installed for the docker steps (e.g.,
pip install prefect-docker
). Helpful docs: - Prefect YAML and templating: prefect.yaml guide - Create deployments: Create deployments - Pull steps (git_clone, set_working_directory): Pull steps API - Docker deployments: Docker infrastructure guide - Work pools and workers: Work pools concept and Workers concept - CLI reference for prefect deploy: prefect deploy CLI If you’d like, share your current prefect.yaml and I can help refactor it to this pattern.
a
@Marvin that's a good start, but it does mean that if I have 10 flows then I actually need to make 20 different deployment configurations. any other options? In my case my build and push are both null and the only differing item is
pull
m
thought for 355.8 seconds
Totally get it. If the only difference is the pull, you’ve got a few options to avoid doubling every deployment. Quick clarification: do your dev and prod use different work pool types (e.g., local vs docker)? If yes, you’ll still need two deployments per flow (a deployment is bound to one work pool). If the pool is the same and just the source differs, you can keep one deployment per flow with a conditional pull step. Options: 1) One deployment per flow with a conditional pull step (no duplication) - Write a tiny custom pull step that checks an env var and either uses the local working directory or clones from git. - Then every flow can share the same deployment; switching behavior is just setting an env var on the worker. Example:
Copy code
# steps/conditional_pull.py
import os
from typing import Optional, Dict
from prefect.deployments.steps.pull import git_clone

async def conditional_pull(
    repository: Optional[str] = None,
    branch: Optional[str] = None,
    commit_sha: Optional[str] = None,
    directory: str = ".",
    env_var: str = "PREFECT_LOCAL_DEV",
) -> Dict[str, str]:
    # If set for local dev, just use the working directory
    if os.getenv(env_var, "").lower() in {"1", "true", "yes"}:
        return {"directory": directory}

    # Otherwise clone from the repo
    if repository:
        return await git_clone(repository=repository, branch=branch, commit_sha=commit_sha)

    # Fallback to current directory
    return {"directory": directory}
prefect.yaml:
Copy code
pull:
  - steps.conditional_pull:
      repository: "<https://github.com/your-org/your-repo.git>"
      branch: "main"
      directory: "."
      env_var: "PREFECT_LOCAL_DEV"
- Run local workers with
PREFECT_LOCAL_DEV=1
so they use your local code; prod workers omit it and clone from git. - Docs for pull steps: Pull steps API - YAML guide: prefect.yaml guide 2) Two files, zero duplication of deployments via overlays (only pull differs) - Keep a base file with all deployments. Keep tiny overlays for dev/prod that only set the top-level
pull
. - In CI, merge the base + overlay into a working prefect.yaml before
prefect deploy
. Example:
Copy code
# prefect.base.yaml (only deployments; no pull)
deployments:
  - name: flow-a
    entrypoint: flows/a.py:a
    work_pool: { name: my-pool }
  - name: flow-b
    entrypoint: flows/b.py:b
    work_pool: { name: my-pool }
  # ... 10 flows here

# prefect.overlay.dev.yaml
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: .

# prefect.overlay.prod.yaml
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/your-org/your-repo.git>
      branch: main
Merge and deploy:
Copy code
yq ea '. as $item ireduce ({}; . * $item )' prefect.base.yaml prefect.overlay.dev.yaml > prefect.yaml
prefect deploy --all
- This keeps the deployments list in one place, and only the pull lives in the overlay files. 3) Generate dev/prod deployments with a small Python script - If you’re okay using the Python API, keep a single list of flows and programmatically create both deployments per flow. - This centralizes the “duplication” as a loop, not repeated YAML blocks. - Flow API: Flow API Notes: - There’s no built-in CLI or env var to “disable pull” on an existing deployment; you either define different pull steps, or make them conditional as in option 1. - If dev and prod require different work pools (e.g., local vs docker), you do need two deployments per flow. Options 2 or 3 will still keep maintenance light. If you can confirm whether your work pools differ between dev and prod, I can tailor one of these patterns with exact snippets for your repo structure.
a
@Marvin those options look great, thanks
m
thought for 12.6 seconds
You got it! If you want, share a quick sketch of your repo layout and how you run local workers, and I can turn option 1 (conditional pull) or option 2 (overlay merge) into a ready-to-drop setup for you.