You’re paying the cost of “git + pip” on every run. The fastest way to eliminate that is to run your flow in an environment where code and dependencies are pre-baked and reused across runs. Here are two solid approaches, with concrete steps and examples. Option A (recommended): Switch to a Docker work pool with a prebuilt image - Why: Code and dependencies are baked once into an image; every run just pulls/starts the container. You can version/tag images by commit SHA and deploy instantly when new images are pushed. - High-level steps: 1) Create a Dockerfile for your flow directory and bake requirements + code into the image. 2) Build/push the image from CI whenever main changes (tag with the commit SHA). 3) Create a Docker work pool and start a Docker worker. 4) Update your deployment to remove pull steps and specify the image. Example Dockerfile

FROM python:3.11-slim

# Optional: configure pip for your private index to speed installs and avoid SSL issues
# You can also bake a pip.conf into the image instead of setting env vars at runtime
ENV PIP_INDEX_URL=https://<your-nexus-or-artifactory> \
    PIP_EXTRA_INDEX_URL=https://<your-nexus-or-artifactory> \
    PIP_NO_CACHE_DIR=1 \
    PIP_DEFAULT_TIMEOUT=60

# Create non-root user
RUN useradd -ms /bin/bash prefect

WORKDIR /app

# Copy and install deps first (better Docker layer cache)
COPY integrations/freight_exchange/direct_freight_search/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy your flow code
COPY integrations/freight_exchange/direct_freight_search/ .

ENV PYTHONPATH=/app
USER prefect

Verified CLI for pool/worker

prefect work-pool create my-docker-pool --type docker
prefect worker start --pool my-docker-pool

Updated prefect.yaml (no pull steps; use image)

name: direct_freight_search
prefect-version: 3.4.20

pull: []  # remove git/pip steps

deployments:
- name: handle_freight_search
  entrypoint: freight_search_flow.py:timocom_search_flow
  parameters:
    pg_secret_block: "postgres-secret-eng"
    timocom_secret_block: "timocom-credentials"
    rabbitmq_secret_block: "rabbitmq-secrets-eng"
    ors_server_variable: "ors-server-eng"
    price_calculation_server_variable: "price-calculation-server-eng"
    predictive_scoring_server_variable: "predictive-scoring-server-eng"
    here_maps_api_key_variable: "here-maps-api-key-dfs-eng"
  work_pool:
    name: my-docker-pool
    job_variables:
      image: <http://registry.example.com/prefect/dfs:main-<git-sha-or-version|registry.example.com/prefect/dfs:main-<git-sha-or-version>>
      env:
        PIP_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
        PIP_EXTRA_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
      # optionally persist the pip cache across runs if you use any dynamic installs
      # volumes:
      # - /var/pip-cache:/home/prefect/.cache/pip
  schedules: []

Notes - Tag images immutably (e.g., main-<sha>) to tie each deployment version to a known code revision. - If you need faster image pulls on your nodes, use a local registry mirror and ensure images are pre-pulled by your infra. Option B: Stay on process workers but preinstall everything once - Why: If Docker isn’t an option, you can avoid per-run installs by using a persistent virtualenv and a persistent checkout on the worker machine(s). Two ways to do it: 1) Install your code as a package - Turn your repo (or subdir) into an installable Python package and publish wheels to your private index. - Install that package + dependencies into the worker’s virtualenv once. - Remove all pull steps from the deployment; set

entrypoint

to your installed module path. - When you release, bump the package version; upgrade the worker environment out-of-band (CI/Ansible/etc.). 2) Use a persistent git checkout and PYTHONPATH - Keep a local checkout of your repo on each worker host (updated by a cron/CI job on commit). - Start the process worker inside the virtualenv that has requirements installed once.

- Remove pull steps. Set

work_pool.job_variables.working_dir

to the existing repo path or add that path to

PYTHONPATH

. Example prefect.yaml for process worker with no per-run git/pip

name: direct_freight_search
prefect-version: 3.4.20

pull: []  # no per-run pulls

deployments:
- name: handle_freight_search
  entrypoint: /opt/repos/trucking-prefect-flows/integrations/freight_exchange/direct_freight_search/freight_search_flow.py:timocom_search_flow
  parameters:
    pg_secret_block: "postgres-secret-eng"
    timocom_secret_block: "timocom-credentials"
    rabbitmq_secret_block: "rabbitmq-secrets-eng"
    ors_server_variable: "ors-server-eng"
    price_calculation_server_variable: "price-calculation-server-eng"
    predictive_scoring_server_variable: "predictive-scoring-server-eng"
    here_maps_api_key_variable: "here-maps-api-key-dfs-eng"
  work_pool:
    name: proc-pool
    job_variables:
      env:
        PYTHONPATH: "/opt/repos/trucking-prefect-flows/integrations/freight_exchange/direct_freight_search"
        # keep your existing env vars as needed
        PIP_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
        PIP_EXTRA_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
      working_dir: "/opt/repos/trucking-prefect-flows/integrations/freight_exchange/direct_freight_search"
  schedules: []

Other incremental speedups (if you must keep git/pip temporarily) - Reuse a persistent directory between runs: - Instead of

git_clone

to a temp dir each run, manage a shared checkout on disk (e.g.,

/opt/repos/...

) updated out-of-band. Then drop

git_clone

and just point

set_working_directory

to the shared path. - Make pip faster: - Ensure a persistent pip cache: set

PIP_CACHE_DIR

to a durable path (or mount it if using Docker). Avoid

PIP_NO_CACHE_DIR=1

. - Host prebuilt wheels for your platform in your private index. Avoid building from source during runs. - If feasible in your environment, consider replacing

pip

with a faster resolver/installer in your images (baked once), then avoid per-run installs entirely. - SSL/trust: Rather than setting

trusted-host

per run, bake your enterprise CA into the image or system trust store so TLS to Artifactory/Nexus succeeds without flags. Docs and references - Docker worker (how-to): Docker worker guide - Workers overview: Workers concept - Customize job variables: Job variables - Deployment steps (pull, git_clone, pip_install_requirements): - Pull steps API - Utility steps API - Deploy CLI: prefect.yaml and CLI deploy Questions to tailor the recommendation - Can your worker environment run Docker containers? If yes, I’d strongly recommend Option A. - If you must stay with process workers, do you have control over the hosts to maintain a persistent venv and repo checkout updated by CI/cron? - Are you already publishing internal Python packages to Nexus/Artifactory? If so, packaging your flow code as a versioned wheel is a great fit for process workers.

Short answer: you don’t need to deploy from inside the worker. Deploy from your dev machine, but make your deployment’s entrypoint a path that exists locally and use the worker’s

working_dir

to point at the persistent checkout on the worker at runtime. Why your deploy failed -

prefect deploy

validates the

entrypoint

path on the machine where you run the command. If you set an absolute path that only exists on the worker (e.g.,

/opt/repos/...

), the deploy from your dev machine will fail because that path doesn’t exist locally. How to deploy cleanly with process workers (no per‑run git/pip) 1) Ensure your workers have a persistent checkout and venv (updated by CI/cron), e.g. on each worker: - Repo at

/opt/repos/trucking-prefect-flows

- Requirements installed once in the worker’s venv 2) In your prefect.yaml: - Remove all

pull

steps - Set the

entrypoint

to a RELATIVE path that exists on your dev machine (and in the worker’s checkout) - Set

work_pool.job_variables.working_dir

to the worker’s repo directory so the runtime can find the same relative entrypoint Example prefect.yaml

name: direct_freight_search
prefect-version: 3.4.20

pull: []  # no per-run git/pip

deployments:
- name: handle_freight_search
  entrypoint: integrations/freight_exchange/direct_freight_search/freight_search_flow.py:timocom_search_flow
  parameters:
    pg_secret_block: "postgres-secret-eng"
    timocom_secret_block: "timocom-credentials"
    rabbitmq_secret_block: "rabbitmq-secrets-eng"
    ors_server_variable: "ors-server-eng"
    price_calculation_server_variable: "price-calculation-server-eng"
    predictive_scoring_server_variable: "predictive-scoring-server-eng"
    here_maps_api_key_variable: "here-maps-api-key-dfs-eng"
  work_pool:
    name: proc-pool
    job_variables:
      # This path exists on the worker host (persistent checkout)
      working_dir: "/opt/repos/trucking-prefect-flows"
      env:
        # Only needed if your flow does runtime pip installs; otherwise can be removed
        PIP_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
        PIP_EXTRA_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
  schedules: []

Deploy from your dev machine - Run the command from your repo’s root so the relative

entrypoint

exists locally:

cd /path/to/trucking-prefect-flows
prefect deploy --prefect-file path/to/freight_search_deployment.yaml

At runtime - The process worker sets its CWD to

working_dir

and uses the relative

entrypoint

to import your flow from the persistent checkout. No git clone, no pip install on each run. Tips and common pitfalls - Keep the same repo layout locally and on the worker. Relative paths make deployments portable. - If you must use an absolute entrypoint, it must exist on the machine running

prefect deploy

. That usually doesn’t match the worker path, so prefer relative paths +

working_dir

. - You can set

working_dir

at the work pool default so you don’t repeat it per deployment (via the pool’s base job template in the UI). - If you ever need imports across the repo, set

PYTHONPATH

in

job_variables.env

to include the repo root. Docs for reference - Workers overview: Workers concept - prefect.yaml and CLI deploy: prefect.yaml and CLI deploy - Customize job variables: Job variables If you share the worker’s checkout path and your local repo structure, I can sanity-check the exact

entrypoint

and

working_dir

values.

You’re hitting a Windows-vs-Linux path issue because the deployment you created points to an absolute Windows path. You don’t need to deploy from inside the worker; instead, deploy from your dev machine with a RELATIVE entrypoint and set the worker’s working_dir to the Linux path that exists on your workers. Do this: 1) Ensure your workers have a persistent checkout - Example path on workers:

/opt/repos/trucking-prefect-flows

(and your venv has the deps installed once). 2) In prefect.yaml, use a relative entrypoint and set working_dir

name: direct_freight_search
prefect-version: 3.4.20

pull: []  # no per-run git/pip

deployments:
- name: handle_freight_search
  entrypoint: integrations/freight_exchange/direct_freight_search/freight_search_flow.py:timocom_search_flow
  parameters:
    pg_secret_block: "postgres-secret-eng"
    timocom_secret_block: "timocom-credentials"
    rabbitmq_secret_block: "rabbitmq-secrets-eng"
    ors_server_variable: "ors-server-eng"
    price_calculation_server_variable: "price-calculation-server-eng"
    predictive_scoring_server_variable: "predictive-scoring-server-eng"
    here_maps_api_key_variable: "here-maps-api-key-dfs-eng"
  work_pool:
    name: proc-pool
    job_variables:
      working_dir: "/opt/repos/trucking-prefect-flows"
      env:
        PIP_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
        PIP_EXTRA_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
  schedules: []

3) Deploy from your dev machine from the repo root - Important: run the command from the project root so the relative entrypoint exists locally.

cd C:\Users\liliya.angelova\repositories\trucking-prefect-flows
prefect deploy --prefect-file path\to\freight_search_deployment.yaml

- When prompted “pull your flow code from a remote storage location?”, answer “n”. The critical piece is that your

entrypoint

stays relative in the YAML; at runtime the process worker will set CWD to

/opt/repos/trucking-prefect-flows

and import the relative entrypoint there. 4) Verify the deployment’s entrypoint is relative

prefect deployment inspect "timocom_search_flow/handle_freight_search"

Confirm that

entrypoint

is

integrations/.../freight_search_flow.py:timocom_search_flow

(not a Windows absolute path). Common pitfalls and fixes - If you pass an absolute path to

prefect deploy

, it will be stored as such and break on Linux. Always use a relative entrypoint. - Keep the same repo layout on dev and workers so the relative path resolves correctly. - Optionally set the

working_dir

at the work pool level (base job template in the UI) so you don’t repeat it per deployment. References - Workers overview: Workers concept - prefect.yaml and CLI deploy: prefect.yaml and CLI deploy - Customize job variables: Job variables If you share your repo root on Linux (e.g.,

/opt/repos/trucking-prefect-flows

) I can sanity-check the exact

entrypoint

and

working_dir

values for you.

You’re very close. There are two things tripping you up: 1) Project root vs. relative entrypoint - When you run with

--prefect-file integrations/.../direct_freight_search/freight_search_deployment.yaml

, Prefect treats that directory as the project root for resolving the

entrypoint

. - Since your prefect.yaml lives alongside

freight_search_flow.py

, your

entrypoint

should be relative to that folder, not the repo root. - Either: - Keep the YAML in that subfolder and set:

entrypoint: freight_search_flow.py:timocom_search_flow

Then run:

prefect deploy --prefect-file .\integrations\freight_exchange\direct_freight_search\freight_search_deployment.yaml

- Or move

prefect.yaml

to the repo root and keep:

entrypoint: integrations/freight_exchange/direct_freight_search/freight_search_flow.py:timocom_search_flow

Then run from the repo root:

prefect deploy

2) Set working_dir to the repo root on the worker - Your current

working_dir

points at the subdirectory. Since your entrypoint includes those subfolders, this can lead to path mismatches. Point the worker to the repo root so a relative entrypoint resolves cleanly. - Change:

work_pool:
    name: proc-pool
    job_variables:
      working_dir: "/home/transmetrics/prefect-w/prefect-repo/trucking-prefect-flows"
      env:
        PIP_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
        PIP_EXTRA_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"

Optional but robust: add a pull step to set CWD on the worker - This locks in the working directory at runtime even if the pool defaults change.

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: "/home/transmetrics/prefect-w/prefect-repo/trucking-prefect-flows"

About the Windows path message - That prompt always echoes the absolute path on the machine running

prefect deploy

. It’s informational. What actually matters at runtime is: - Your

entrypoint

(keep it relative), and - The worker’s working directory (set to the Linux path). - Verify what’s stored with:

prefect deployment inspect "timocom_search_flow/handle_freight_search"

Ensure

entrypoint

is relative (no C:\ path). Quick corrected examples A) YAML stays in the subfolder

name: direct_freight_search
prefect-version: 3.4.20

pull: []
deployments:
- name: handle_freight_search
  entrypoint: freight_search_flow.py:timocom_search_flow
  parameters:
    pg_secret_block: "postgres-secret-eng"
    timocom_secret_block: "timocom-credentials"
    rabbitmq_secret_block: "rabbitmq-secrets-eng"
    ors_server_variable: "ors-server-eng"
    price_calculation_server_variable: "price-calculation-server-eng"
    predictive_scoring_server_variable: "predictive-scoring-server-eng"
    here_maps_api_key_variable: "here-maps-api-key-dfs-eng"
  work_pool:
    name: proc-pool
    job_variables:
      working_dir: "/home/transmetrics/prefect-w/prefect-repo/trucking-prefect-flows"
      env:
        PIP_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
        PIP_EXTRA_INDEX_URL: "{{ prefect.blocks.secret.nexus-url }}"
  schedules: []

Run:

prefect deploy --prefect-file .\integrations\freight_exchange\direct_freight_search\freight_search_deployment.yaml

B) YAML is at the repo root

entrypoint: integrations/freight_exchange/direct_freight_search/freight_search_flow.py:timocom_search_flow
work_pool.job_variables.working_dir: "/home/transmetrics/prefect-w/prefect-repo/trucking-prefect-flows"

Run from repo root:

prefect deploy

If you still see a failure at runtime, please paste the worker log (the traceback and the exact path it tries to open). I’ll pinpoint whether it’s a duplicated path issue or something else. References: - Workers concept: Workers concept

- prefect.yaml and CLI deploy: prefect.yaml and CLI deploy - Pull steps (set_working_directory): Pull steps API