Adrien Besnard
08/13/2025, 1:02 PMprepare_flow_for_deployment
within a CLI to deploy some flows. How to avoid the attempted relative import beyond top-level package
issue if my flow imports some stuff from other relative packages?Marvin
08/13/2025, 1:02 PMMarvin
08/13/2025, 1:07 PMrepo/
pyproject.toml
src/
my_project/
__init__.py
flows/
__init__.py
my_flow.py # from my_project.lib.utils import helper
lib/
__init__.py
utils.py
- Then import absolutely in your flow: from my_project.lib.utils import helper
- Ensure the runtime can import your package:
- In local/dev: pip install -e .
- In containers: add a build step or Dockerfile line to pip install -e .
- Don’t use “upward” relative imports
- from ..lib import utils
will fail if the module is at the top of the package. Keep imports within the package or switch to absolute imports.
- If you can’t (yet) package: set PYTHONPATH to your repo root
- Set PYTHONPATH
in the environment where your flow runs (worker/job/container) so import my_project...
works.
- As a last-resort hack in your entrypoint before imports:
import sys, pathlib
sys.path.append(str(pathlib.Path(__file__).resolve().parents[2])) # repo root
But really, packaging is cleaner and far less brittle.
Notes on Prefect versions:
- If you’re on Prefect 3.x (recommended): use prefect deploy
or programmatic flow.from_source(...).deploy(...)
with an entrypoint like src/my_project/flows/my_flow.py:my_flow
. Your imports should be absolute and your package installed or on PYTHONPATH.
- If you’re on Prefect 2.x (where prepare_flow_for_deployment
lives): the same Python rules apply. You’ll still want a proper package and absolute imports, or a correctly set PYTHONPATH in the execution environment.
If you share:
- Your Prefect version (2.x or 3.x)
- A small snippet of your project tree
- The exact import that’s failing
…I can give you the most direct, “change-these-lines” fix.Adrien Besnard
08/13/2025, 1:37 PMMarvin
08/13/2025, 1:37 PMMarvin
08/13/2025, 1:40 PMprefect deploy my_project.flows.my_flow:my_flow
Notes:
- Your project must be a real package (with init.py files) and importable (e.g., pip install -e .).
- Use absolute imports in your flow code: from my_project.lib.utils import helper
.
- Programmatically (3.x)
from prefect.flows import Flow
# source='.' can be your project root or remote storage
flow = Flow.from_source('.', 'my_project.flows.my_flow:my_flow')
flow.deploy(name='my-deployment', work_pool_name='process-pool')
- In prefect.yaml (3.x)
deployments:
- name: my-deployment
entrypoint: my_project.flows.my_flow:my_flow
If you’re still on Prefect 2.x (where prepare_flow_for_deployment
lives), the same idea applies: pass a module path where an entrypoint is expected and ensure your package is importable. But I’d recommend moving to 3.x and prefect deploy
or Flow.from_source(...).deploy(...)
anyway.
Want me to sanity check your layout and exact command? Tell me your Prefect version and show a quick tree of your repo plus the failing import. I’ll point at the precise line to change — with minimal sighing.Adrien Besnard
08/13/2025, 1:42 PMMarvin
08/13/2025, 1:42 PMMarvin
08/13/2025, 1:43 PMprefect deploy my_project.flows.my_flow:my_flow
- Python
from prefect.flows import Flow
Flow.from_source(
source='.', # your project root (or an absolute path)
entrypoint='my_project.flows.my_flow:my_flow',
).deploy(name='my-deployment', work_pool_name='process-pool')
Why '.': it tells Prefect “don’t pull code from elsewhere; import it as-is.” Since your package is already on PYTHONPATH, Prefect will import my_project.flows.my_flow:my_flow
directly.
Caveats (because of course there are):
- If the worker runs on a different machine/container, make sure that environment also has your package installed (or PYTHONPATH set). '.' on that machine won’t magically contain your code.
- If you do want Prefect to pull code remotely instead, set source to a repo URL (e.g., a Git URL) and keep the module-path entrypoint.Adrien Besnard
08/13/2025, 1:48 PMMarvin
08/13/2025, 1:48 PMMarvin
08/13/2025, 1:49 PMfrom pathlib import Path
from prefect.flows import Flow
Flow.from_source(
source=str(Path("/abs/path/to/your/project")), # not '.'
entrypoint="my_project.flows.my_flow:my_flow",
).deploy(name="my-deployment", work_pool_name="process-pool")
2) Remote workers or containers (Docker/K8s/etc.)
One of:
- Install your package into the image/env
- Dockerfile:
COPY pyproject.toml .
COPY src/ src/
RUN pip install -e .
- Then you can use:
Flow.from_source(source=".", entrypoint="my_project.flows.my_flow:my_flow")
Because the package is installed, the module path resolves anywhere.
- Or ship code via a source that the worker can pull
- Git:
Flow.from_source(
source="<https://github.com/your-org/your-repo.git@main>",
entrypoint="my_project.flows.my_flow:my_flow",
)
Prefect will clone the repo for the run, set the working directory, then import your module.
- Remote storage (S3/GCS/Azure)
- Create the storage block, point source at it, keep the module-path entrypoint. Prefect pulls the code then imports it.
3) If you insist on PYTHONPATH
- Set PYTHONPATH in the worker’s environment (work pool base job template or deployment env) to include your project root. Then use:
prefect deploy my_project.flows.my_flow:my_flow
- Do not rely on your local shell’s PYTHONPATH; the worker needs it.
Quick sanity checks
- In the worker/container, run:
python -c "import my_project; from my_project.flows.my_flow import my_flow; print(my_flow)"
If that fails, Prefect won’t be able to import it either.
- Make sure my_project
is a real package (has init.py).
If you tell me:
- Where the worker runs (local process, Docker, K8s)
- How you’re deploying (CLI vs Flow.from_source)
- Your current source value and entrypoint
…I’ll point at the exact knob to twist. With only minimal eye-rolling.Adrien Besnard
08/13/2025, 1:51 PMMarvin
08/13/2025, 1:51 PMMarvin
08/13/2025, 1:52 PMfrom prefect.flows import Flow
Flow.from_source(
source="/home/adrien/Work/your-project", # project root directory
entrypoint="my_package/flows/my_flow.py:my_flow", # note the .py
).deploy(name="my-deployment", work_pool_name="process-pool")
- CLI:
prefect deploy ./my_package/flows/my_flow.py:my_flow
Option B — module path entrypoint
- Only if your package is importable (installed or PYTHONPATH set in the worker).
Flow.from_source(
source=".", # any directory; not used for pulling when imports work
entrypoint="my_package.flows.my_flow:my_flow",
)
And/or:
prefect deploy my_package.flows.my_flow:my_flow
Common gotchas that cause your exact error:
- File path style without “.py” (e.g., my_package/flows/my_flow:my_flow
) → Prefect treats it as a module name and tries to import “my_package/flows/my_flow” literally.
- Absolute filesystem path used in a module-style entrypoint (e.g., /home/.../my_flow.py:my_flow
but parsed as a module).
- Worker env can’t import your package (PYTHONPATH/installation missing).
If you paste your exact Flow.from_source(...)
or prefect deploy ...
command and your project root path, I’ll point to the one thing to change.Adrien Besnard
08/13/2025, 2:02 PMMarvin
08/13/2025, 2:03 PMMarvin
08/13/2025, 2:04 PMfrom prefect.flows import load_flow_from_entrypoint
flow = load_flow_from_entrypoint("my_project.flows.my_flow:my_flow")
flow.deploy(name="my-deployment", work_pool_name="process-pool")
- File path (no package install needed, but path must end with .py)
import os
from prefect.flows import load_flow_from_entrypoint
os.chdir("/home/adrien/Work/your-project") # ensure cwd is the project root
flow = load_flow_from_entrypoint("src/my_project/flows/my_flow.py:my_flow")
flow.deploy(name="my-deployment", work_pool_name="process-pool")
Common pitfalls (the ones that bite)
- Mixing styles: "my_project/flows/my_flow:my_flow" (no .py) looks like a file path but gets treated like a module — leading to “No module named 'my_project/flows/my_flow'”.
- Using an absolute filesystem path in a module-style entrypoint. If it’s a path, include .py.
- The worker/container can’t import your package (PYTHONPATH not set, or not installed). Module form won’t work there until you fix the environment.
If you share the exact entrypoint string you’re passing to load_flow_from_entrypoint and where your code runs (local vs Docker/K8s), I’ll tell you the one change to make.