You’re hitting the classic “src/ layout” issue: locally your shell likely has PYTHONPATH set (or you’re running from the project root), but your Prefect worker’s runtime doesn’t know about your

src/

directory. You have two good options. The first is the most robust. 1) Best practice: install your project as a package in the worker environment - Add a minimal pyproject that tells pip your code lives under src:

[project]
  name = "my_project"
  version = "0.1.0"
  dependencies = ["prefect>=3"]

  [tool.setuptools.packages.find]
  where = ["src"]

- Then ensure your worker environment installs it: - Process worker on a VM: run

pip install -e .

(or

pip install .

) in the same environment before starting

prefect worker start

. - Docker/Kubernetes: install in the image (e.g.,

COPY . /app

then

pip install -e /app

). - With the package installed,

import my_project...

will work without tweaking PYTHONPATH. 2) Quick fix: put src on PYTHONPATH for the run - Set

PYTHONPATH

for the deployment via job variables so the runtime sees `./src`: - In your `prefect.yaml`:

name: my-project
    prefect-version: 3.0.0
    deployments:
      - name: my-flow
        entrypoint: flows/my_flow.py:my_flow
        work_pool:
          name: process-pool
        job_variables:
          env:
            PYTHONPATH: "${PYTHONPATH}:./src"

- Or set default env on the work pool (UI or CLI base job template) to include

PYTHONPATH

. - Docker: in your Dockerfile

WORKDIR /app
    ENV PYTHONPATH="/app/src:${PYTHONPATH}"

- Docs on job variables: Customize job variables - Docs on settings and env behavior: Settings and profiles About .env files - Prefect will load a

.env

in the working directory locally, but remote workers won’t see it unless it’s present in the runtime environment. Don’t rely on

.env

for

PYTHONPATH

unless you commit it and it’s available where the worker executes. Which option to pick? - If you control the environment (VM, Docker, K8s), installing your project as a package is the most reliable and avoids fragile path hacks. - If you need something fast, set

PYTHONPATH

via deployment

job_variables.env

or the work pool’s default env. Quick question to tailor the exact fix: which worker type are you using (process on a VM, Docker, Kubernetes), and are you deploying from a git source (

flow.from_source(...).deploy(...)

) or running locally with a process worker on the same machine as your repo?

Got it — thanks for the context. Since you’re running a process worker inside the official Prefect image, your code isn’t baked into the container and Prefect will clone your repo at run time. With a src/ layout, you must ensure the flow run process adds ./src to sys.path. If both “install the package” and “set PYTHONPATH” didn’t work, we likely have a config mismatch. Let’s nail it down. First, a quick checklist - Are you deploying from git with

flow.from_source(..., entrypoint="src/my_project/...:my_flow")

? - In your deployment, is

job_variables.env.PYTHONPATH

set to

src

(not

${PYTHONPATH}:./src

)? Using

${PYTHONPATH}:...

won’t expand if PYTHONPATH is unset. - Does your repo have

src/my_project/__init__.py

? (Namespace packages can work, but having init.py removes ambiguity.) - Is your flow run actually seeing the env you set? Minimal working deployment (process worker + from_source + src) Make sure your deployment has these parts (either in Python or prefect.yaml). Example prefect.yaml:

name: my-project
prefect-version: 3.0.0
deployments:
  - name: my-flow
    entrypoint: src/my_project/flows.py:my_flow
    work_pool:
      name: process-pool
    # Prefect clones your repo and sets CWD to the repo root for the run
    # Add src to sys.path via PYTHONPATH
    job_variables:
      env:
        PYTHONPATH: "src"

Docs: Customize job variables Debug quickly in a failing run Add this to the very top of your flow file (temporarily) to see what the process actually sees:

import os, sys
print("CWD:", os.getcwd())
print("PYTHONPATH:", os.environ.get("PYTHONPATH"))
print("Has ./src:", os.path.isdir("src"))
print("First 5 sys.path:", sys.path[:5])

You should see: - CWD is the clone root of your repo - Has ./src: True - PYTHONPATH: src If “Has ./src” is False, your working directory isn’t the repo root; share your deployment config so we can adjust the pull/working_dir. If PYTHONPATH is empty, your

job_variables.env

didn’t apply — share your prefect.yaml or how you call

.deploy(...)

and we’ll correct the structure. If you’d prefer a bulletproof route Bake your code into the image and skip PYTHONPATH entirely: - Dockerfile

FROM prefecthq/prefect:3.0-python3.11
  WORKDIR /opt/prefect
  COPY . /opt/prefect
  RUN pip install --no-cache-dir -e .
  # start your process worker in this image
  CMD ["prefect", "worker", "start", "-p", "process-pool"]

- Start this image as your worker container. Since

my_project

is installed into site-packages,

import my_project

will work regardless of the checkout path. Next step Could you paste: - Your

prefect.yaml

(or

.deploy(...)

call) - How you start the worker container (docker run command) - The short debug output (CWD/PYTHONPATH/Has ./src) With that, I can pinpoint exactly why the env isn’t taking effect and get you unblocked.

Thanks for sharing the snippet — a couple things stand out that likely explain why your env didn’t take effect: 1) Your YAML structure looks off - The deployment block must live under a top-level

deployments:

key, not under

definitions:

. - If the block is in the wrong section, Prefect will ignore it and your

PYTHONPATH

won’t be applied. Here’s a minimal corrected skeleton you can try:

name: my-project
prefect-version: 3.0.0

definitions:
  work_pools:
    default_work_pool: &default_work_pool
      name: default
      work_queue_name: null
      job_variables:
        env:
          # keep it simple; do not rely on ${PYTHONPATH} expansion
          PYTHONPATH: "src"

deployments:
  - name: Test setup
    entrypoint: src/athena_data_pipeline/flows/main.py:test_setup
    work_pool: *default_work_pool
    schedules: []
    # optionally override at deployment level to be sure it’s applied
    job_variables:
      env:
        PYTHONPATH: "src"

Docs for job variables: Customize job variables 2) Ensure your working directory is the repo root when the flow runs With a process worker in a container, Prefect will clone your repo at run time. Python’s

sys.path[0]

will be the working directory of the process. If the CWD is not the repo root,

PYTHONPATH: "src"

won’t resolve. If you’re using Prefect Projects with

prefect deploy

, add pull steps so the working directory is set to the cloned repo root:

pull:
  - prefect.deployments.steps.git_clone:
      repository: "<your repo URL>"
      branch: "main"
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ clone_dir }}"

Then the

PYTHONPATH: "src"

will work because CWD == repo root. 3) Quick runtime debug Add this at the top of your flow file temporarily to confirm what the run sees:

import os, sys
print("CWD:", os.getcwd())
print("PYTHONPATH:", os.environ.get("PYTHONPATH"))
print("Has ./src:", os.path.isdir("src"))
print("sys.path[:5]:", sys.path[:5])

You should see: - CWD is the repo root - PYTHONPATH is “src” - Has ./src: True If CWD isn’t the repo root, we need to adjust the pull steps or set

job_variables.working_dir

appropriately. 4) If you want a bulletproof solution that doesn’t rely on CWD Since you’re running the worker inside the official Prefect image, bake your code into that image and install it as a package: - Dockerfile for the worker container:

FROM prefecthq/prefect:3.0-python3.11
WORKDIR /opt/prefect
COPY . /opt/prefect
RUN pip install --no-cache-dir -e .
CMD ["prefect", "worker", "start", "-p", "default"]

When your code is installed into site-packages,

import my_project

works regardless of working directory or PYTHONPATH. 5) Last-resort in-code path adjustment If you prefer not to modify YAML or the image, add this once near the top of your flow module to add the absolute src path:

import sys, pathlib
src_dir = pathlib.Path(__file__).resolve().parents[2]  # repo/src
sys.path.insert(0, str(src_dir))

In your layout repo/src/athena_data_pipeline/flows/main.py,

parents[2]

points to the absolute “src” folder. If this still doesn’t work, could you: - Confirm whether you’re using

prefect deploy

with pull steps, or

flow.from_source(...).deploy(...)

? - Share the full

prefect.yaml

(with pull steps if present) - Paste the small debug output (CWD/PYTHONPATH/Has ./src) That’ll let me pinpoint exactly why

PYTHONPATH

isn’t being applied by the process worker in your container.

Perfect — then set the working directory explicitly to that subdirectory for the run, and keep PYTHONPATH relative to it. The most reliable way (with a process worker pulling code at run time) is to use pull steps in prefect.yaml: - Clone the repo - Set the working directory to your defined subdirectory - Keep PYTHONPATH set to src so imports like import my_project work Example prefect.yaml

name: my-project
prefect-version: 3.0.0

# Pull steps run on the worker before your flow starts
pull:
  - prefect.deployments.steps.git_clone:
      repository: "<https://github.com/your-org/your-repo.git>"
      branch: "main"
      # token, submodules, etc. if needed
  - prefect.deployments.steps.set_working_directory:
      # Point this at the subdirectory you want as CWD
      directory: "{{ clone_dir }}/path/to/your/subdir"

definitions:
  work_pools:
    default_work_pool: &default_work_pool
      name: default
      job_variables:
        env:
          # PYTHONPATH is now relative to the subdirectory above
          PYTHONPATH: "src"

deployments:
  - name: Test setup
    # Entry point is relative to the working_dir set above
    entrypoint: src/athena_data_pipeline/flows/main.py:test_setup
    work_pool: *default_work_pool
    schedules: []

Notes - Entry point is resolved relative to the working directory you set. - With CWD set to your subdir, PYTHONPATH: "src" makes your src/ package importable. Why not just job_variables.working_dir? - You can set

job_variables.working_dir

, but it must be an absolute path. Since the repo is cloned to a dynamic path at runtime, you won’t know that absolute path ahead of time. The pull step

set_working_directory

is designed for this and uses the

{{ clone_dir }}

value provided by the clone step. Quick sanity check in your flow (temporarily)

import os, sys
print("CWD:", os.getcwd())
print("PYTHONPATH:", os.environ.get("PYTHONPATH"))
print("Has ./src:", os.path.isdir("src"))
print("sys.path[:5]:", sys.path[:5])

You should now see CWD at your defined subdir and PYTHONPATH=src. Docs: - Pull steps and job variables: Customize job variables - Prefect YAML overview: prefect.yaml If you share the exact subdirectory you need, I can plug it into a ready-to-run YAML for you.