<@ULVA73B9P> is there a way to have global definit...
# ask-marvin
f
@Marvin is there a way to have global definition.yaml file which the definitions are shared across multiple prefect.yaml files?
m
thought for 305.2 seconds
Short answer: not natively. In Prefect 3.x there isn’t a supported “global definitions.yaml” you can import or include across multiple
prefect.yaml
files. Each project has its own
prefect.yaml
, and there’s no built-in include/import mechanism. Options to reduce duplication: - Within a single file: use YAML anchors/aliases to reuse fragments in that same
prefect.yaml
. - Centralize infra in Prefect itself: put shared config into workspace resources (e.g., work pools and blocks) and reference them by name in each project’s
prefect.yaml
. This keeps YAML light and the real “shared config” lives in Prefect. - Use environment variables/templating: Prefect 3 supports Jinja templating in
prefect.yaml
, so you can inject env vars or computed values. Combine with profiles/`.env` files to standardize values across projects. - Monorepo: keep a single
prefect.yaml
at the repo root and define multiple deployments there. - Generate config: keep a shared template and render per-project
prefect.yaml
in CI (e.g., a small script or Cookiecutter) before running deploys. - Define deployments in Python with shared helpers: create a small internal package that encapsulates your common defaults and call it from each repo. Example of a shared Python helper for consistent deployments:
Copy code
# common_deploy.py (in an internal package)
from prefect import flow

def deploy_standard(*, repo, entrypoint, name, work_pool="default", tags=None, parameters=None):
    return flow.from_source(
        source=repo,
        entrypoint=entrypoint,
    ).deploy(
        name=name,
        work_pool_name=work_pool,
        tags=tags or [],
        parameters=parameters or {},
    )

# In each repo:
# from common_deploy import deploy_standard
# deploy_standard(
#     repo="<https://github.com/acme/my-service>",
#     entrypoint="flows/my_flow.py:my_flow",
#     name="my-flow",
#     work_pool="acme-prod-pool",
#     tags=["prod"],
# )
Docs and references: - prefect.yaml docs - Settings and profiles - Managing Prefect deployments with Terraform (if you want to manage deployments centrally via infra-as-code) If you can share what you’re trying to “define once” (e.g., work pool config, image tags, actions/steps, env vars) and whether you’re in a mono- or multi-repo setup, I can suggest a concrete pattern.
f
@Marvin tell me more about this - Use environment variables/templating: Prefect 3 supports Jinja templating in
prefect.yaml
, so you can inject env vars or computed values.
m
thought for 134.6 seconds
Great question! Here’s how templating and env vars work in Prefect 3’s prefect.yaml. What you can template - Most string fields in prefect.yaml can be Jinja-templated: deployment names, tags, version, work pool name, job_variables, action commands, etc. - Templating is rendered when Prefect reads prefect.yaml (e.g., during prefect deploy), not at flow-run time. Available template context - env: your process environment at deploy time. - Examples: -
{{ env.MY_VAR }}
to require a var -
{{ env.get('MY_VAR', 'default') }}
for a default - steps: outputs from earlier build/push actions (by id). Useful for using command outputs (e.g., git SHA) later in the file. Common patterns and examples 1) Inject env vars into deployment fields
Copy code
deployments:
  - name: "my-flow-{{ env.get('ENV', 'dev') }}"
    entrypoint: flows/my_flow.py:my_flow
    tags:
      - "{{ env.get('ENV', 'dev') }}"
      - "team-{{ env.get('TEAM', 'data') }}"
    work_pool:
      name: "{{ env.get('POOL', 'default') }}"
    job_variables:
      image: "<http://ghcr.io/acme/my-image:{{|ghcr.io/acme/my-image:{{> env.get('IMAGE_TAG', 'latest') }}"
      env:
        APP_ENV: "{{ env.get('ENV', 'dev') }}"
        LOG_LEVEL: "{{ env.get('LOG_LEVEL', 'INFO') }}"
2) Reuse outputs from earlier steps (e.g., git SHA) across the file
Copy code
build:
  - id: git_sha
    run: git rev-parse --short HEAD
  - run: docker build -t <http://ghcr.io/acme/app:{{|ghcr.io/acme/app:{{> steps.git_sha.stdout }} .

push:
  - run: docker push <http://ghcr.io/acme/app:|ghcr.io/acme/app:>{{ steps.git_sha.stdout }}

deployments:
  - name: "my-flow"
    entrypoint: flows/my_flow.py:my_flow
    version: "{{ steps.git_sha.stdout }}"
    work_pool:
      name: "prod-pool"
    job_variables:
      image: "<http://ghcr.io/acme/app:|ghcr.io/acme/app:>{{ steps.git_sha.stdout }}"
3) Lists and maps can contain templated strings
Copy code
deployments:
  - name: "batch-{{ env.get('ENV', 'dev') }}"
    tags:
      - "{{ env.get('ENV', 'dev') }}"
      - "{{ env.get('TEAM', 'data-eng') }}"
    job_variables:
      env:
        DATASET: "{{ env.get('DATASET', 'default') }}"
        REGION: "{{ env.get('REGION', 'us-east-1') }}"
Notes and limitations - Rendering time: Templates are evaluated when you run Prefect commands that read the file (e.g., prefect deploy). They are not re-evaluated at flow-run time. If you want values to vary per run, pass them via parameters, or set env in the worker/job environment. - Data types: Jinja expressions are embedded in YAML strings, which Prefect then parses. Simple values work best; avoid trying to build complex Python objects in templates. - Secrets: Be careful injecting secrets via env at deploy time; they’ll be rendered on your machine. Prefer storing credentials in Prefect Blocks or use job/infra env provided to the worker runtime. - Where env vars come from: Prefect reads your process environment. If you keep values in a .env file, make sure they’re loaded into the shell before running prefect deploy (e.g., direnv, dotenv, or your CI’s env configuration). Optional: Prefect Variables - Prefect Variables are workspace-scoped values you can manage via CLI or UI and read inside flows. If you want to centralize some values there: - Set a variable:
Copy code
prefect variable set APP_ENV prod
- Get it later in code:
Copy code
from prefect.variables import get
    env = await get("APP_ENV")  # in async context
- Whether Variables can be referenced directly in templated prefect.yaml depends on your workflow and security posture; many teams prefer pulling them at runtime inside the flow or injecting them as job env via your worker configuration. Docs - Prefect YAML: prefect.yaml docs - Customizing job variables (e.g., image, env): Customize job variables - Settings and profiles (where Prefect reads env/config): Settings and profiles
- Variables concept: Variables If you share what you want to template (e.g., image tags, pool names, schedules, parameters), I can sketch a precise snippet tailored to your setup.