You’re running into a real/expected behavior: in Prefect 3.x,

Flow.from_source(...).deploy(...)

will clone your repo onto the worker at run time, but it will not pip-install your project. With a src-layout package, importing your package will fail unless you either (a) install it or (b) put the src directory on

PYTHONPATH

. Here are three patterns that work well on-prem, with process workers and private repos (no Docker): 1) Set PYTHONPATH so your src layout is importable (no install) - Keep your

entrypoint

pointing to the file in

src/...

. - Ensure the worker’s job has

PYTHONPATH

include the cloned repo’s

src

directory and run with the repo root as the working directory. - Do this by passing job variables on deploy:

from prefect import Flow
from prefect.storage import GitRepository, GitCredentials

Flow.from_source(
    source=GitRepository(
        url="<ssh://git@your.git.host/your-org/your-repo.git>",
        credentials=GitCredentials(ssh_private_key="...or a saved Block..."),
        branch="main",
    ),
    entrypoint="src/fpga_automation/server/flows/kasli_release.py:kasli_soc_release_flow",
).deploy(
    name="production",
    work_pool_name=work_pool,
    parameters=params,
    tags=["fpga", "kasli_soc", "production"],
    description="Production deployment for Kasli-SoC FPGA builds",
    job_variables={
        # Process worker honors these
        "working_dir": "{{ clone-directory }}",
        "env": {
            # Add 'src' to Python path so your package is importable without install
            "PYTHONPATH": "{{ clone-directory }}/src:${PYTHONPATH}"
        },
    },
    version=PACKAGE_VERSION,
)

Notes: -

{{ clone-directory }}

is populated by Prefect’s git pull step at run time. - This avoids pip-installing your package, which is nice on HPC/process workers. 2) Install your package on the worker at run time (editable or regular install) - If you prefer standard package imports, you can instruct the worker to pip-install your repo after clone and before import. - Add pull/setup steps via a

prefect.yaml

at your repo root (it’s picked up when you deploy/run from source) to run a small install script:

# prefect.yaml
pull:
  - prefect.deployments.steps.git_clone:
      repository: <ssh://git@your.git.host/your-org/your-repo.git>
      branch: main
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ clone-directory }}"
  - prefect.deployments.steps.utility.run_shell_script:
      script: |
        python -m pip install --upgrade pip
        python -m pip install -e .

- This gives you consistent “code = installed package” semantics without pre-provisioning. Use SSH or a token via

GitCredentials

for private repos. 3) Pre-provision the worker environment (what you’re doing now) - This is still a valid pattern for process workers. The server and worker venvs do not need to be identical; only the worker needs the run-time deps. - If you keep it, consider distributing a single built venv (or conda env) to your worker nodes, or pin a wheels/lockfile you install from, to keep it less “hacky”. Which to choose? - Fastest change: Option 1 (set

PYTHONPATH

and working dir). No install needed; works great for src-layout. - Most “package-correct”: Option 2 (pip install at run time). Ensures imports behave like they do locally, and aligns the installed package version with the checked-out commit. - Operationally simple at scale: Option 3 (pre-provisioned env), if you already have a good way to fan out consistent environments. A few extra tips: - Private repos: use

GitCredentials

with SSH keys or PATs stored as Prefect Secrets/Blocks so workers can clone securely. - Entry point path: your

entrypoint="src/...py:flow_func"

is fine for src-layout; just make sure you either install the package or expose

src

on

PYTHONPATH

.

- Process worker variables: the process worker honors

env

and

working_dir

in

job_variables

so you can control PYTHONPATH and working directory per deployment. - If you can flatten your repo (put

fpga_automation/

at repo root instead of under

src/

), Python will find it without `PYTHONPATH`—but I realize many teams prefer src-layout. Docs for deeper reference: - Storing and pulling flow code from source: Store flow code (Prefect 3) - Deployments and work pools overview: Deployments concept If you can share whether you’d prefer “no install” (PYTHONPATH) or “install at run time,” I can tailor an exact

prefect.yaml

or

deploy(...)

snippet for your setup.

Totally fair concern. A few clarifications that might help: - prefect.yaml is optional. You can keep everything in Python. If you do prefer YAML, it lives alongside your code (commit it at your repo root). There’s no chicken-and-egg at run time: the pull steps defined by the YAML are stored on the Prefect server as part of the deployment. At flow run time, the worker executes those stored steps; it does not need to “read” the YAML from your repo to know what to do. If you want to avoid YAML entirely, here’s the simplest Python-only fix for src-layout packages: - Keep using Flow.from_source with your GitRepository and entrypoint under

src/...

- Set

working_dir

to the clone directory and add

src

to PYTHONPATH via

job_variables

in your deploy call Example:

from prefect import Flow
from prefect.storage import GitRepository, GitCredentials

Flow.from_source(
    source=GitRepository(
        url="<ssh://git@your.git.host/your-org/your-repo.git>",
        branch="main",
        credentials=GitCredentials(ssh_private_key="** or use a saved Block **"),
    ),
    entrypoint="src/fpga_automation/server/flows/kasli_release.py:kasli_soc_release_flow",
).deploy(
    name="production",
    work_pool_name="your-process-work-pool",
    parameters=params,
    tags=["fpga", "kasli_soc", "production"],
    description="Production deployment for Kasli-SoC FPGA builds",
    version=PACKAGE_VERSION,
    job_variables={
        "working_dir": "{{ clone-directory }}",
        "env": {
            "PYTHONPATH": "{{ clone-directory }}/src:${PYTHONPATH}"
        },
    },
)

-

{{ clone-directory }}

is populated by Prefect when it clones your repo for the run. This avoids any pip install and works well on on-prem/process workers. - For private repos, pass

GitCredentials

(SSH key or token) or a saved Block. If you do want the “install my package after clone” behavior, using a small

prefect.yaml

in the repo root is the cleanest way (the server stores the resulting steps; the worker doesn’t need the YAML at run time):

# prefect.yaml (commit at repo root)
pull:
  - prefect.deployments.steps.git_clone:
      repository: <ssh://git@your.git.host/your-org/your-repo.git>
      branch: main
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ clone-directory }}"
  - prefect.deployments.steps.utility.run_shell_script:
      script: |
        python -m pip install --upgrade pip
        python -m pip install -e .

Either approach keeps everything centralized in your repo and works with process workers, on-prem, and private Git. Refs: - Customize job variables (env, working_dir): Customize job variables - Storing/pulling flow code from source (incl. private Git): Store flow code from source If you prefer to stay Python-only, I’d go with the

PYTHONPATH

+

working_dir

approach above. Want me to adapt that snippet to your exact work pool name and credentials setup?

Short answer: the replacement happens during step templating, not inside the git clone logic itself. Pointers to the exact places in Prefect 3.x source: - Where the templating/replacement happens - Step runner applies templating to each step’s inputs before execution (this is where

target_directory

or similar placeholders get rendered): - prefect/deployments/steps/core.py (see

run_step

/

run_steps

) - The templating engine that actually performs the substitution: - prefect/utilities/templating.py (see

apply_values

, plus helpers like

find_placeholders

,

resolve_variables

, and

resolve_block_document_references

) - Where git clone uses the rendered value - The

git_clone

step receives the already-rendered arguments (including any clone/target directory) and performs the clone: - prefect/deployments/steps/pull.py (see

git_clone

) Related change introducing an explicit

target_directory

argument for `git_clone`: - PR: init: add target_directory to git_clone If you want me to point to exact line numbers, let me know the Prefect version/tag you’re on (e.g., 3.1.9) and whether you mean the general templating pass (in

core.py

) or a specific

git_clone

argument like

target_directory

.

You’re right — there’s no

clone-directory

template in Prefect 3.x work pool base job templates. That looks historical from older patterns (e.g., Projects/agents era). In 3.x: - Work pool base job templates expose variables like

working_dir

,

env

,

command

, etc. (what you showed). There’s no

clone-directory

there. - The clone destination is controlled in your deployment steps using the

git_clone

step’s

target_directory

, then you can set the process working directory either via a subsequent step or by templating your deployment’s

work_pool.job_variables.working_dir

. Practical example (prefect.yaml):

pull:
  - prefect.deployments.steps.git_clone:
      id: clone
      repository: <https://github.com/owner/repo.git>
      branch: main
      target_directory: ./app-src

  # Option A: set working directory via a step
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ clone.directory }}"

# Option B: set working_dir on the job from the step output
work_pool:
  name: my-pool
  job_variables:
    working_dir: "{{ clone.directory }}"

Notes: - The

git_clone

step takes

target_directory

(added in this PR: init: add target_directory to git_clone). - The

set_working_directory

step just switches to a directory for subsequent steps/flow execution. - Step outputs can be referenced by giving a step an

id

(e.g.,

id: clone

) then using

{{ clone.<key> }}

in later steps or job variables. The

git_clone

step returns the path it cloned into under a key you can reference as shown above. Source pointers (where this is implemented): - Step execution and templating: prefect/deployments/steps/core.py (run_step/run_steps) - Templating engine (placeholder replacement): prefect/utilities/templating.py (

apply_values

,

resolve_variables

, etc.) - Git clone and working dir steps: prefect/deployments/steps/pull.py (

git_clone

,

set_working_directory

) - API reference for these steps: Prefect 3 API: deployments.steps.pull If you share what you were trying to set with

clone-directory

, I can translate it to the 3.x pattern with the exact YAML you’ll need.

Great question. In Prefect 3.x the clean way to support a src layout is to set

PYTHONPATH

via your deployment’s job variables (so the flow run process sees it), ideally alongside

working_dir

. Two common patterns: 1) Set working_dir to the repo root and use a relative PYTHONPATH

pull:
  - prefect.deployments.steps.git_clone:
      id: clone
      repository: <https://github.com/owner/repo.git>
      branch: main
      target_directory: ./repo

work_pool:
  name: process-pool
  job_variables:
    working_dir: "{{ clone.directory }}"
    env:
      PYTHONPATH: src

2) Use an absolute PYTHONPATH if you don’t set working_dir

pull:
  - prefect.deployments.steps.git_clone:
      id: clone
      repository: <https://github.com/owner/repo.git>
      target_directory: ./repo

work_pool:
  name: process-pool
  job_variables:
    env:
      PYTHONPATH: "{{ clone.directory }}/src"

Notes: - The

git_clone

step returns the clone path as `{{ clone.directory }}`; you can reference it in

job_variables

. - Relative

PYTHONPATH

entries are resolved against the

working_dir

of the flow run process. If you set

working_dir

to the repo root,

PYTHONPATH: src

is sufficient. - Avoid relying on a shell

export

inside a step — it won’t persist into the flow run process. Use

job_variables.env

instead. Alternative to PYTHONPATH: - If you prefer, install your package so imports work without PYTHONPATH:

pull:
  - prefect.deployments.steps.git_clone:
      id: clone
      repository: <https://github.com/owner/repo.git>
  - prefect.deployments.steps.utility.run_shell_script:
      script: "pip install -e {{ clone.directory }}"

Refs: - Steps API (git_clone, set_working_directory): https://github.com/PrefectHQ/prefect/blob/main/src/prefect/deployments/steps/pull.py - Step execution/templating: https://github.com/PrefectHQ/prefect/blob/main/src/prefect/deployments/steps/core.py - Templating engine: https://github.com/PrefectHQ/prefect/blob/main/src/prefect/utilities/templating.py - Prefect YAML guide: https://docs-3.prefect.io/v3/how-to-guides/deployments/prefect-yaml If you share your worker type (Process, Docker, Kubernetes) and repo shape, I can tailor the exact

prefect.yaml

snippet.

You’re right —

{{ ... }}

templating is only available in YAML step definitions. When using the Python API, set the env directly on the deployment’s job variables. For a src layout, the simplest pattern is to set PYTHONPATH to point at

src

. If you’re using

Flow.from_source(...)

, the repo is cloned for the flow run and the process’s working directory will be that clone root unless you’ve overridden it — so a relative

src

works well. Example with the Python API:

from prefect.flows import Flow

flow = Flow.from_source(
    source="<https://github.com/owner/repo.git>",
    entrypoint="flows/my_flow.py:flow",
)

flow.deploy(
    name="prod",
    work_pool_name="my-pool",
    job_variables={
        "env": {
            "PYTHONPATH": "src"   # relative to the working directory of the flow run
        }
    },
)

If you’ve set a custom working directory or you’re on Docker/Kubernetes and want to be explicit, use an absolute path inside the runtime environment:

flow.deploy(
    name="prod",
    work_pool_name="my-pool",
    job_variables={
        "working_dir": "/opt/prefect/app",           # where your code will live at runtime
        "env": {"PYTHONPATH": "/opt/prefect/app/src"}
    },
)

Alternatives if you’d rather not rely on PYTHONPATH: - Make your project installable and importable without PYTHONPATH (e.g.,

pyproject.toml

+

pip install -e .

). For Python-API deployments you’d typically bake this into the runtime image (Docker/K8s) or ensure the environment where the worker runs has your package installed. If you share: - whether you’re using

Flow.from_source(...)

or deploying from a local entrypoint, and - your worker type (Process, Docker, Kubernetes), I can tailor the exact

job_variables

you should use.

Short answer: PYTHONPATH is used by the Python interpreter at process startup. Prefect does not “special-handle” it, but whether it takes effect depends on where you set it and what the job’s working directory is at the time the Python process starts. Relative PYTHONPATH entries are supported by Python, but they’re resolved relative to the process’s working directory at startup, which may not be what you expect in different worker types. A few things to check and clarify so I can help precisely: - Which worker type are you using (Process, Docker, Kubernetes)? - Where are you setting PYTHONPATH (deployment env, work pool/job template env, the machine’s shell that launches the worker, inside your code)? - Does the import failure happen when loading your flow file (import-time) or later during task execution? - Can you capture sys.path at the very top of your entrypoint before any of your project imports? Why your logs can look “right” but imports still fail: - os.environ['PYTHONPATH'] only shows the current environment variable. Python resolves PYTHONPATH into sys.path when the interpreter starts; changing the env later won’t retroactively change sys.path. - If you’re logging after imports (e.g., inside your flow function), the failure may already have occurred. - With relative PYTHONPATH, if the job’s working directory isn’t where you think it is when the Python process starts, the path resolution won’t match your repo layout even if os.getcwd() later looks correct. Concrete debugging steps 1) At the very top of your entrypoint (before importing your own modules), print these:

import os, sys
print("cwd:", os.getcwd())
print("PYTHONPATH:", os.environ.get("PYTHONPATH"))
print("sys.path:", sys.path)

This confirms the working directory and what Python actually uses for imports. 2) If needed, force the path early (workaround):

import os, sys
sys.path.insert(0, os.path.abspath("src"))  # or the exact absolute path to your code

3) Ensure the env var is set where the run actually happens: - Process worker: set env on the deployment or on the work pool’s base job template. The worker spawns a subprocess for the run; it inherits the job env. - Docker/Kubernetes: env must be in the container/pod that runs the flow. Set it in the work pool’s base job template or the deployment so it’s injected into the job spec. - Note OS path separator: use ':' on Linux/macOS, ';' on Windows. 4) Prefer absolute paths over relative PYTHONPATH Relative PYTHONPATH is brittle because the working directory can differ by worker type/config. Absolute paths are reliable. 5) Best practice Avoid PYTHONPATH when you can: - Make your repo a proper Python package and install it into the environment used by the run (e.g., in your Docker image or via your job’s build/install step). - Or use a from_source deployment that checks out your repo and then installs your package before running. If you share: - Worker type - How/where PYTHONPATH is being set - A snippet of your entrypoint with the early logging (cwd, PYTHONPATH, sys.path) - The exact import that fails and your repo layout …I can pinpoint the cause and suggest the cleanest fix for your setup.