<@ULVA73B9P> I have a `kubernetes` work pool and I...
# ask-marvin
t
@Marvin I have a
kubernetes
work pool and I have a script that calls
prefect --no-prompt deploy --all --prefect-file /tmp/prefect.yaml
. The thing is the
prefect.yaml
is generated from a template which generates the
deployments
list with
job_variables.image
being overwritten to external docker images and the
entrypoints
defined are not accessible from the CLI caller. This leads to errors while deploying those files:
Copy code
FileNotFoundError: [Errno 2] No such file or directory:
How can I define the deployments in such a way the CLI doesn’t try to reach to the file I’m specifying in the
deployments[].entrypoint
?
m
thought for 1065.5 seconds
Short answer: with Prefect 3.x,
prefect deploy
must import the flow specified by
deployments[].entrypoint
at deploy time to read metadata and build the parameter schema. There isn’t a CLI flag to skip that import. That’s why you’re seeing a FileNotFoundError when the entrypoint file isn’t present where the CLI runs. You’ve got a few workable patterns: 1) Check out or mount the code before calling
prefect deploy
- Keep file-path entrypoints like
flows/my_flow.py:my_flow
- Ensure the repo is available in the environment that runs the CLI:
Copy code
git clone <https://github.com/your-org/your-repo> /tmp/project
cd /tmp/project
prefect --no-prompt deploy --all --prefect-file prefect.yaml
- This is the simplest if your flows live in a repo 2) Use module-path entrypoints and install your flow package in the deploy environment - Switch to a module path entrypoint (still imported at deploy time, but no file path):
Copy code
deployments:
  - name: my-deployment
    entrypoint: my_package.flows.my_flow
    work_pool:
      name: my-k8s-pool
      job_variables:
        image: <http://ghcr.io/org/app:tag|ghcr.io/org/app:tag>
- Ensure the environment running
prefect deploy
can import
my_package
(e.g.,
pip install my_package==X.Y.Z
) - This is a good fit if your flow code is packaged and baked into the runtime image 3) Create deployments programmatically from a remote source (no local files) - Use the Python API to pull code from Git and deploy:
Copy code
from prefect.flows import Flow

Flow.from_source(
    source="<https://github.com/your-org/your-repo.git>",
    entrypoint="flows/my_flow.py:my_flow",
).deploy(
    name="my-deployment",
    work_pool_name="my-k8s-pool",
    job_variables={"image": "<http://ghcr.io/org/app:tag|ghcr.io/org/app:tag>"},
    build=False,
    push=False,
)
- This validates against the remote repo and avoids needing local files when creating the deployment Important notes -
pull
steps in
prefect.yaml
(e.g.,
git_clone
) run on the worker at flow run time, not during
prefect deploy
. They won’t prevent the CLI from importing your entrypoint at deploy time. - Entrypoint formats: - File path:
path/to/flow.py:flow_func
(requires file at deploy time) - Module path:
package.module.flow_func
or
package.module:flow_func
(requires the package to be importable at deploy time) - There is no
--skip-import
or similar flag for
prefect deploy
. Docs - Prefect YAML reference: prefect.yaml - Storing/pulling flow code (git, storage, etc.): Store flow code - Customizing job variables (e.g., setting the Kubernetes image): Customize job variables If you share how your code is distributed (repo vs only in images) and where you’re running the deploy command (CI, local, etc.), I can suggest the cleanest option for your setup.
t
@Marvin the code is distributed only in images
m
thought for 102.2 seconds
Got it — if your flow code exists only inside container images, the machine running
prefect deploy
won’t be able to import your entrypoints. In Prefect 3.x the CLI always imports the flow at deploy time; there’s no flag to skip that. You have three viable patterns: Option A (recommended): Run deploy inside the same image that contains the flow - Build/push the image(s) that contain the flow code. - For each image, run a short container to execute the deploy command from inside that image so the entrypoint import succeeds. - Example (Docker):
Copy code
docker run --rm \
  -e PREFECT_API_URL=$PREFECT_API_URL \
  -e PREFECT_API_KEY=$PREFECT_API_KEY \
  -v $PWD/prefect.yaml:/app/prefect.yaml \
  <http://ghcr.io/your-org/your-image:tag|ghcr.io/your-org/your-image:tag> \
  bash -lc 'prefect --no-prompt deploy --all --prefect-file /app/prefect.yaml'
- If you have multiple images (each with different flows), run one deploy job per image. Use
--name
pattern(s) to target just the deployments for that image if needed. Option B: Use module-path entrypoints with a tiny “stub” package locally - Convert entrypoints to module paths (e.g.,
my_app.flows.orders:run
or
my_app.flows.orders.run
). - Create a minimal local package that defines stub flows matching those names so
prefect deploy
can import them. - Set
enforce_parameter_schema: false
in each deployment to avoid schema mismatches between stubs and real flows. - At runtime, the worker uses your container image (with the real code) and imports the real flow. - Example snippet:
Copy code
deployments:
  - name: orders-etl
    entrypoint: my_app.flows.orders:run
    enforce_parameter_schema: false
    work_pool:
      name: my-k8s-pool
      job_variables:
        image: <http://ghcr.io/your-org/orders:2024-11-23|ghcr.io/your-org/orders:2024-11-23>
Note: This works, but maintaining stubs can be brittle if real flow signatures change. Option A is usually cleaner. Option C: Programmatic deploy inside the image - Instead of the CLI, run a small script inside the image that imports the flow and calls
flow.deploy(...)
. This still requires running inside the image that has the code, similar to Option A. Runtime considerations for image-only code - If you use file-path entrypoints (e.g.,
flows/orders.py:run
) in your deployments, set the runtime working directory to where the code lives in the container:
Copy code
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /app
deployments:
  - name: orders-etl
    entrypoint: flows/orders.py:run
    work_pool:
      name: my-k8s-pool
      job_variables:
        image: <http://ghcr.io/your-org/orders:2024-11-23|ghcr.io/your-org/orders:2024-11-23>
- Module-path entrypoints don’t need a working directory, but the package must be on PYTHONPATH in the image. Why
pull
steps don’t help here -
pull
steps (like
git_clone
) execute on the worker at flow run time, not during
prefect deploy
. They won’t prevent the CLI from trying to import your entrypoint at deploy time. Docs - Prefect YAML reference: prefect.yaml - Store/pull flow code: Store flow code - Customize job variables (set Kubernetes image): Customize job variables If you share how many images you have and whether you can run a CI matrix, I can outline a concrete deploy plan (one job per image vs. a single run with stubs).