<@ULVA73B9P> How to deploy flow using python scrip...
# ask-marvin
r
@Marvin How to deploy flow using python script. Flow source is placed in python virtual environment.
m
thought for 418.8 seconds
Great question. Assuming you’re on Prefect 3.x, here are the main ways to create a deployment with a Python script when your flow code is already available in your virtual environment. Key idea: your deployment just needs a resolvable entrypoint to your flow at runtime. You can reference your flow either by: - Python module path (when your code is installed in the environment), or - File path within a source directory (local dir or Git) Below are concise, working patterns. Option A — Your flow is installed in the virtualenv (recommended when workers have the same package installed) - Prefect can infer a Python-module entrypoint when you deploy directly from an imported flow. - Ensure the worker environment has the same package installed. Example:
Copy code
from yourpkg.flows import etl_flow  # your flow is installed and importable

# Creates a deployment that imports "yourpkg.flows:etl_flow" at run time
etl_flow.deploy(
    name="etl-prod",
    work_pool_name="process-pool",
    tags=["prod"]
)
When to use: your workers run in environments where
yourpkg
is installed (e.g., Docker image, VM, or venv) and can import your flow. Option B — Your flow lives in a local directory on disk - Use
Flow.from_source(...)
to point to a directory and a file-based entrypoint like
./path/to/file.py:flow_function
. - This works only if the worker can see the same path (e.g., same machine or shared filesystem). Example:
Copy code
from prefect import Flow

Flow.from_source(
    source=".",  # project root on disk
    entrypoint="flows/etl.py:etl_flow"
).deploy(
    name="etl-local",
    work_pool_name="process-pool"
)
Option C — Source is in Git (great when workers run remotely) - Reference a Git repo and the same file-based entrypoint. - For private repos, configure credentials via environment/blocks. Example:
Copy code
from prefect import Flow

Flow.from_source(
    source="<https://github.com/org/repo>",   # or git@github.com:org/repo.git
    entrypoint="flows/etl.py:etl_flow"
).deploy(
    name="etl-git",
    work_pool_name="process-pool"
)
Starting a worker - Make sure a worker is polling your work pool; otherwise scheduled runs won’t get picked up. - Verified command:
Copy code
prefect worker start -p process-pool
Notes and gotchas - Installed-code approach (Option A): no extra storage is needed, but every worker that executes your flow must be able to
import
the module (same package/version). - Local-path approach (Option B): only works if the worker has the same filesystem path available. If your worker runs elsewhere (e.g., k8s, VM), use Git (Option C) or package your code into the worker’s image/environment. - For Docker/Kubernetes, you can set
image=
on
deploy()
and control
build/push
, but that’s optional if you’re not containerizing. If you prefer the CLI - From a project directory, you can do:
Copy code
prefect deploy ./flows/etl.py:etl_flow -n etl-prod -p process-pool
This expects a file-based entrypoint in a Prefect project directory. A couple of clarifying questions to tailor this to you: - Will your worker run on the same machine as your local virtualenv? - Is your flow code installed as a package (e.g.,
pip install -e .
) in the worker environment, or do you need to pull from Git?
r
@Marvin Clarifying: 1. worker is on the same machine as virtualenv 2. flow code is installed as package using pip I have tried option A from your answer, but it ends with error : ValueError: Either an image or remote storage location must be provided when deploying a deployment. I do not want to use image or remote storage.
m
thought for 50.2 seconds
Thanks for the details — that explains the error. What’s happening - When you call
your_flow.deploy(...)
directly (Option A), Prefect uses a Python-module entrypoint (e.g.,
yourpkg.flows:etl_flow
). For module-based entrypoints, Prefect requires a code distribution mechanism. That’s why you see: “ValueError: Either an image or remote storage location must be provided…”. - Since you don’t want an image or remote storage, the fix is to deploy with a local source directory (on disk) so the worker reads the code directly from the filesystem. Since your worker is on the same machine AND the code is installed in the same virtualenv, you can point Prefect at the installed package directory in site-packages. Do this instead (Option B using your installed package’s folder as the source):
Copy code
import importlib
from pathlib import Path
from prefect import Flow

# Replace 'yourpkg' and entrypoint below to match your package structure
pkg = importlib.import_module("yourpkg")
pkg_root = Path(pkg.__file__).parent
# e.g. .../venv/lib/python3.x/site-packages/yourpkg

Flow.from_source(
    source=pkg_root,
    entrypoint="flows/etl.py:etl_flow"  # relative to pkg_root
).deploy(
    name="etl-local",
    work_pool_name="process-pool",
    build=False,   # ensure no image build attempt
    push=False     # ensure no push attempt
)
How to pick the entrypoint string - It’s the file path (relative to
pkg_root
) plus
:function_name
of the flow. - Examples: - If your flow is defined in
yourpkg/flows.py
as
etl_flow
, use
entrypoint="flows.py:etl_flow"
. - If it’s in
yourpkg/flows/etl.py
as
etl_flow
, use
entrypoint="flows/etl.py:etl_flow"
. Start a worker for your pool
Copy code
prefect worker start -p process-pool
Why this works - Prefect now treats the installed package directory as a local source of truth. Since your worker is on the same machine/venv, it can read those files directly—no image and no remote storage needed. If you want, share: - The module path to your flow function (e.g.,
yourpkg.flows:etl_flow
) - The package layout (where the file actually lives) I can give you the exact
entrypoint
string to use.
r
@Marvin I do not want to use exact path to flow in virtual environment. It does not work, because I am using uv package manager and flow is installed as editable module. But command line prefect deploy works, when prefect.yaml looks like this : pull: - prefect.deployments.steps.set_working_directory: directory: "." deployments: - name: LIF_0197 entrypoint: lif_0197.main:lif_0197 lif_0197 is name of module with flow. So CLI command "prefect deploy" works as I need, but I want do same using python code. It should be definitely possible, because CLI command is using python code too.
m
thought for 62.0 seconds
You’re right — the CLI succeeds because it uses the project pipeline (prefect.yaml) which sets a local working directory as “storage”. When you call
flow.deploy(...)
with a module entrypoint, Prefect requires an image or remote storage. To replicate the CLI behavior in Python without image/remote storage, construct a RunnerDeployment with LocalStorage and then call the top-level deploy function. Do this:
Copy code
from pathlib import Path
from prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment
from prefect.storage import LocalStorage

# Use the project root as your working directory, just like your prefect.yaml
project_root = Path(".").resolve()

deployment = RunnerDeployment(
    name="LIF_0197",
    entrypoint="lif_0197.main:lif_0197",  # module:func, same as your prefect.yaml
    work_pool_name="process-pool",
    storage=LocalStorage(path=project_root),  # local working dir; no image/remote storage
    # optional: tags=[], parameters={}, description="...", etc.
)

# Create/update the deployment in Prefect
deploy(deployment, build=False, push=False)
Why this works -
LocalStorage(path=...)
mirrors your
prefect.yaml
pull step (
set_working_directory: "."
), telling the worker to run from that directory. - With a module entrypoint, Prefect accepts a local storage location (working directory) instead of an image or remote storage, as long as the worker is on the same machine and can import the module (your uv editable install handles that). Start the worker
Copy code
prefect worker start -p process-pool
Notes - Keep
build=False, push=False
to avoid image building/pushing. - You don’t need to reference the virtualenv’s site-packages path; pointing storage to your project root (".") is enough, matching the CLI behavior you already have. - If your worker starts from a different CWD, prefer an absolute path for
LocalStorage(path=...)
as shown above.
r
@Marvin I am getting error : ModuleNotFoundError: No module named 'prefect.storage' Please fix it.
m
thought for 96.1 seconds
Thanks for the catch — that import path was my mistake. Some Prefect 3.x versions don’t expose
prefect.storage
the way I suggested. Use this instead (mirrors your CLI prefect.yaml with set_working_directory="."):
Copy code
from prefect import Flow

Flow.from_source(
    source=".",                           # same as set_working_directory: "."
    entrypoint="lif_0197.main:lif_0197"   # module:function, same as your CLI
).deploy(
    name="LIF_0197",
    work_pool_name="process-pool",
    build=False,
    push=False
)
Notes: - Run this from your project root so
source="."
points to the correct working directory. - This avoids any image or remote storage; your worker is on the same machine and can import the editable module via uv, so it will work just like the CLI. - Start your worker if it’s not already running:
Copy code
prefect worker start -p process-pool
Docs for reference: - Deployments overview - prefect deploy CLI
r
@Marvin It fails with error : ValueError: Either an image or remote storage location must be provided when deploying a deployment. We have already discussed it. CLI must use something else.
m
thought for 45.6 seconds
You’re right — the CLI is attaching a local working directory as storage, which my prior snippet didn’t replicate. Sorry about the bad import earlier. Here’s the corrected Python equivalent of your prefect.yaml + prefect deploy flow, without image or remote storage:
Copy code
from pathlib import Path
from prefect.deployments import deploy
from prefect.deployments.runner import RunnerDeployment
from prefect.runner.storage import LocalStorage  # correct import

project_root = Path(".").resolve()

deployment = RunnerDeployment(
    name="LIF_0197",
    entrypoint="lif_0197.main:lif_0197",   # module:function, same as your CLI
    work_pool_name="process-pool",
    storage=LocalStorage(path=str(project_root)),  # mirrors set_working_directory: "."
)

# Register/update the deployment; no image and no remote storage
deploy(deployment, build=False, push=False)
Then start your worker:
Copy code
prefect worker start -p process-pool
Why this fixes the error: - A module-based entrypoint without storage requires either an image or a storage location. - The CLI sets a working directory via the pull step; the above code does the same by providing
LocalStorage
, so Prefect does not require an image or remote storage. References: - Deployments overview - prefect deploy CLI If you still hit issues, please share your project root and the exact
lif_0197
layout; I’ll tailor the
entrypoint
and
LocalStorage
path precisely.