Short answer: make the runtime cwd match how your entrypoints are written, or make your entrypoints independent of cwd. In Prefect 3.x there are a few good patterns for this. Key points - Entrypoints like

src/my_app/flows/etl.py:flow

are resolved at runtime relative to the working directory of the process that executes the flow. - For the Process work pool, you can set the working directory that the flow subprocess will use with the

working_dir

job variable. Options to fix the mismatch 1) Set a working_dir for your Process work pool (recommended) - Put the correct path to your project root on the bare metal host into the deployment’s job variables so that relative entrypoints resolve the same as on your dev machine. Example

prefect.yaml

snippet:

deployments:
  - name: etl-prod
    entrypoint: src/my_app/flows/etl.py:flow
    work_pool:
      type: process
      name: baremetal-pool
      job_variables:
        working_dir: "{{ env.PREFECT_WORKDIR | default('/opt/prefect/my_app') }}"

- You can set this once as the pool default so you don’t repeat it in every deployment: 1) Generate the base template:

prefect work-pool get-default-base-job-template --type process > process.json

2) Edit

process.json

so

"variables.properties.working_dir.default"

is your desired path. 3) Apply it:

prefect work-pool update "baremetal-pool" --base-job-template process.json

This command and the working_dir field are supported; see the default template output for Process pools which includes `working_dir`:

prefect work-pool get-default-base-job-template --type process

Docs: - Customize job variables - prefect.yaml guide 2) Use absolute entrypoint paths - If the install location on the host is stable, you can make the entrypoint absolute to avoid cwd dependence:

entrypoint: /opt/prefect/my_app/src/my_app/flows/etl.py:flow

- You can still parameterize with env vars:

entrypoint: "{{ env.PREFECT_ENTRYPOINT | default('/opt/prefect/my_app/src/my_app/flows/etl.py:flow') }}"

3) Parameterize per environment via env interpolation - Prefect 3.x supports interpolation in

prefect.yaml

. A common pattern is to template the working_dir and keep entrypoints relative:

work_pool:
  type: process
  name: baremetal-pool
  job_variables:
    working_dir: "{{ env.PREFECT_WORKDIR }}"

Then set

PREFECT_WORKDIR

differently on each host. 4) Package your code and avoid file paths (optional) - If you install your flows as a Python package on the host, you can avoid deep relative paths and rely on a stable install location plus

PYTHONPATH

if needed. You can set env in job variables:

job_variables:
  env:
    PYTHONPATH: "/opt/prefect/my_app"

A note if you later run in containers (Podman/Docker) - The Docker worker template doesn’t have a dedicated

working_dir

field, but you can set it via

container_create_kwargs

or by structuring your command. You’d also ensure the host path is volume-mounted. Happy to share a Podman-friendly example if you head that direction. Questions to help tailor this: - Are you using a Process work pool on that bare metal host? - What path does your project live at on the host (the path you want as cwd)? - Do you want a single

prefect.yaml

to work for both dev and prod, or are you okay with prod-only values? If you share your current

prefect.yaml

snippet and the host’s folder layout, I can propose an exact patch.

Short answer: they’re not the same, and they take effect at different times. - job_variables.working_dir - Applies when the worker creates the execution environment (infra-level). - For Process workers, it sets the CWD of the subprocess that will run your flow. - For container workers, the CWD generally comes from the container image’s WORKDIR (job variables for “working_dir” may not exist for that worker type). - Think of this as: “where does the process/container start?” - pull: set_working_directory - Runs inside the runtime right before your flow is imported. - It changes directories after any code has been pulled/cloned so imports and relative paths resolve from that directory. - Works consistently across worker types (process and containers). - Think of this as: “where should the flow import and run from now that the code is here?” Why the proposed pull step helps - It’s worker-agnostic and happens after your code is available. If you’re pulling/cloning code at runtime (e.g., from Git or remote storage),

set_working_directory

ensures the flow runs from the cloned/pulled directory even if the initial working directory was different. - With containers (Docker/Podman/Kubernetes), the initial CWD is usually set by the image; the pull step lets you cd into the actual repo/project path where the code landed. - If both are used, the pull step runs last and determines the final CWD that your imports and relative paths will use. Important caveat for “local filepaths” - Neither setting mounts host directories into a container. If you need to access host paths from a container, you still need volume mounts. The working directory only affects relative paths within the runtime’s filesystem. If you can share which worker type you’re using (Process vs Docker/Podman vs Kubernetes), I can tailor the exact recommendation. For Process workers,

job_variables.working_dir

can be sufficient if your code is already on disk. If you clone/pull code at runtime or run in containers, adding

pull: set_working_directory

is the more reliable, portable way to ensure relative paths and imports work. References: - prefect.yaml and pull steps - Pull steps API (set_working_directory) - Process worker job variables (working_dir) - Customize job variables

Great — since you’re on a process worker and all flows need access to local file paths and a shared Conda environment, here’s a clean, repeatable prefect.yaml you can use. I’ll show two variants: - Local filesystem source (code already present on the host) - Git-based source (worker clones your repo at run time) Notes upfront - Assuming Prefect 3.x. Process worker job variables support: name, env, labels, command, stream_output, working_dir. - Start your worker inside your Conda env so all 50–100 flows share it. Example:

conda activate myenv
  prefect worker start --pool rhel-process-pool

- Top-level build/push/pull steps apply to all deployments by default; per-deployment steps override them. - Use absolute paths for working_dir to ensure access to host file paths. Links - prefect.yaml guide - Workers and work pools - Create deployments - Customize job variables - Pull steps API 1) Local filesystem source (recommended for your case) - Code lives on the same RHEL9 host as the worker (e.g., /opt/prefect/project) - No package installs at runtime; rely on your shared Conda env - All deployments share the same work_pool and job_variables via YAML anchors

# Set this to your actual project name and installed Prefect version
name: acme-data-pipelines
prefect-version: 3.0.0  # update to match your installed version

definitions:
  # Reusable defaults for all deployments
  base-work-pool: &base-work-pool
    name: rhel-process-pool
    job_variables:
      # Start the subprocess in your project root so relative imports work
      working_dir: /opt/prefect/project
      stream_output: true
      env:
        # Example: make shared data paths explicit
        DATA_ROOT: /mnt/data
        PREFECT_LOGGING_LEVEL: INFO

# Optional: apply this to all deployments by default; per-deployment sections override it
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/project

deployments:
  - name: ingest-daily
    entrypoint: flows/ingest.py:ingest_flow
    work_pool: *base-work-pool
    tags: [prod, ingest]
    parameters:
      source: "{{ $INGEST_SOURCE | default('<s3://bucket/path>') }}"
    schedules:
      - cron: "0 6 * * *"        # 6AM daily
        timezone: "UTC"

  - name: transform-hourly
    entrypoint: flows/transform.py:transform_flow
    work_pool: *base-work-pool
    tags: [prod, transform]
    schedules:
      - cron: "0 * * * *"        # hourly
        timezone: "UTC"

  # Add more flows below; reuse *base-work-pool
  - name: load-weekly
    entrypoint: flows/load.py:load_flow
    work_pool: *base-work-pool
    tags: [prod, load]
    schedules:
      - cron: "0 3 * * 1"        # Mondays at 03:00 UTC
        timezone: "UTC"

2) Git-based source (if you prefer the worker to clone code each run) - Worker will clone your repo at run time, then run from that clone - Still uses the local Conda env; no Docker/Podman required - Replace the repository URL and optionally add credentials via a block ``` name: acme-data-pipelines prefect-version: 3.0.0 definitions: base-work-pool: &base-work-pool name: rhel-process-pool job_variables: # Use the cloned directory output from the pull step working_dir: "{{ clone-repo.directory }}" stream_output: true env: DATA_ROOT: /mnt/data PREFECT_LOGGING_LEVEL: INFO pull: - prefect.deployments.steps.git_clone: id: clone-repo repository: https://github.com/your-org/your-repo.git branch: main # credentials: "{{ prefect.blocks.github-credentials.my-gh-token }}" # optional - prefect.deployments.steps.set_working_directory:

directory: "{{ clone-repo.directory }}" deployments: - name: ingest-daily entrypoint: flows/ingest.py:ingest_flow work_pool: *base-work-pool tags: [prod, ingest] schedules: - cron: "0 6 * * *" timezone: "UTC" - name: transform-hourly entrypoint: flows/transform.py:transform_flow work_pool: *base-work-pool tags: [prod, transform] schedules: - cron: "0 * * * *" timezone: "UTC"

Create and run your process work pool and worker
- Verified commands:

prefect work-pool create "rhel-process-pool" --type process # Start the worker from your Conda env conda activate myenv prefect worker start --pool rhel-process-pool

Deploy your flows
- From your project root (where prefect.yaml lives):

prefect deploy --all # or deploy a single entrypoint: # prefect deploy flows/ingest.py:ingest_flow --name ingest-daily ``` A few recommendations for your setup - Keep your worker service running inside the shared Conda env to avoid per-run installs. - If some flows need special env vars (e.g., data roots, credentials), prefer setting them in work_pool.job_variables.env to keep them consistent. - For 50–100 flows, continue adding entries under deployments; the YAML anchor keeps duplication low. If you split by repo/module, consider multiple prefect.yaml files per project. Questions to tailor this exactly for you - What is your project root path on the RHEL9 host (e.g., /opt/prefect/project)? - What is the name of the shared Conda environment you use to start the worker? - Do you want the worker to clone your repo each run (git) or rely on a pre-synced local directory? - Any schedules you want pre-baked for these deployments, or do you prefer manual/triggered runs? If you share those details, I’ll fill them into a final prefect.yaml for you.

Short answer: almost. - When you run prefect deploy, the

entrypoint

must be valid relative to your project root (where

prefect.yaml

lives). - At run time, if your

entrypoint

is a relative file path (e.g.,

flows/ingest.py:ingest_flow

), it’s resolved relative to the process’s current working directory. For a process worker, that’s whatever you set as

working_dir

(via

job_variables

or a

set_working_directory

pull step). So in practice: - If

working_dir

=

/opt/prefect/project

and

entrypoint

=

flows/ingest.py:ingest_flow

, the worker will run

/opt/prefect/project/flows/ingest.py

. - If you clone code to a temp dir in a pull step and set

working_dir

to that cloned path, the same

entrypoint

will resolve inside that clone. Tip: You can also use a module path entrypoint like

my_package.flows.ingest:ingest_flow

. That uses Python import semantics and is less sensitive to CWD, as long as your

PYTHONPATH

or

working_dir

puts the package on the path. Docs: prefect.yaml guide

You’ve got a few good options. Pick the one that matches how/where your code lives: 1) Code is already on the worker host (but prefect.yaml is elsewhere) - Set the process worker’s working_dir to the code directory (absolute path). - Keep entrypoints relative to that working_dir. Example:

name: my-project
prefect-version: 3.0.0

definitions:
  base: &base
    name: rhel-process-pool
    job_variables:
      working_dir: /opt/prefect/code-repo   # absolute path to your code
      stream_output: true

deployments:
  - name: ingest-daily
    entrypoint: flows/ingest.py:ingest_flow # resolved under /opt/prefect/code-repo
    work_pool: *base

2) Code lives in Git (not present where you run prefect deploy) - Use pull steps to clone the repo and set working_dir from the clone output. - Entrypoints are relative to the repo root. - If your CLI environment can’t import the flow at deploy time, set

enforce_parameter_schema: false

. Example:

name: my-project
prefect-version: 3.0.0

definitions:
  base: &base
    name: rhel-process-pool
    job_variables:
      working_dir: "{{ clone.directory }}"
      stream_output: true

pull:
  - prefect.deployments.steps.git_clone:
      id: clone
      repository: <https://github.com/your-org/your-repo.git>
      branch: main
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ clone.directory }}"

deployments:
  - name: ingest-daily
    entrypoint: flows/ingest.py:ingest_flow
    work_pool: *base
    # enforce_parameter_schema: false   # use if deploy-time import is not possible

3) Code is installed as a package in your Conda env - Use a module path entrypoint (less sensitive to CWD). - Ensure the package is importable in the worker’s Conda env. Example:

deployments:
  - name: ingest-daily
    entrypoint: acme_pipelines.flows.ingest:ingest_flow
    work_pool:
      name: rhel-process-pool
      job_variables:
        stream_output: true

Rule of thumb - Entrypoints that are file paths are resolved relative to the process’s working directory at run time. For a process worker, set

job_variables.working_dir

(or a

set_working_directory

pull step) to the directory where your code actually lives. - If prefect.yaml is “far” from your code, either point working_dir at the code path or clone the repo at run time, then keep entrypoints relative to that directory. Module path entrypoints also work well if your code is installed. Docs: - prefect.yaml guide - Pull steps API - Process worker job variables If you can share where the code lives on the RHEL9 host (e.g., /srv/pipelines), I’ll tailor the exact snippet for you.

You’re hitting this because

prefect deploy

resolves each entrypoint relative to the project root (the directory containing prefect.yaml), not the worker’s

working_dir

. The

working_dir

only affects runtime on the worker — it does not help the CLI find your flow files at deploy time. Three solid fixes — pick one: 1) Co-locate prefect.yaml with your code (simplest) - Put prefect.yaml in the repo/code root and run

prefect deploy --all

from there. - Keep entrypoints as relative file paths (e.g.,

flows/ingest.py:ingest_flow

). - Keep your process worker’s

working_dir

pointing to the same directory for consistent runtime behavior. 2) Use module-path entrypoints and install your code into the Conda env - Install your code so it’s importable where you run `prefect deploy`:

conda activate myenv
  pip install -e /opt/prefect/code-repo

- Change entrypoints to module paths:

entrypoint: acme_pipelines.flows.ingest:ingest_flow

- Now

prefect deploy --all

can import your flows even if prefect.yaml lives elsewhere. 3) Keep prefect.yaml separate but avoid deploy-time imports - If you cannot make the files visible at deploy time, disable parameter schema import and rely on runtime setup:

deployments:
    - name: ingest-daily
      entrypoint: flows/ingest.py:ingest_flow
      enforce_parameter_schema: false
      work_pool:
        name: rhel-process-pool
        job_variables:
          working_dir: /opt/prefect/code-repo

- This skips importing the flow during deploy (which is where the FileNotFoundError happens). However, the file still needs to exist at runtime in

/opt/prefect/code-repo

. If your code is not guaranteed to be there on the worker, add pull steps (e.g., git_clone) and set

working_dir

from the clone output. Quick checks - From the directory where you run

prefect deploy --all

, does

./flows/ingest.py

actually exist for each entrypoint? If not, use one of the options above. - If you prefer module paths, confirm

python -c "import acme_pipelines"

succeeds in the same Conda env you use for

prefect deploy

. If you share: - The absolute path to your code root on RHEL9 - Where your prefect.yaml currently lives - An example entrypoint that fails …I’ll provide a precise patch to your prefect.yaml and the minimal CLI steps to get

prefect deploy --all

working. Docs: - prefect.yaml guide - Create deployments - Process worker job variables

Short answer: yes, if those absolute paths exist on the machine where you run prefect deploy. Details - prefect deploy needs to load each flow at deploy time. If your entrypoints are absolute file paths like /opt/prefect/project/flows/ingest.py:ingest_flow, the CLI will find them regardless of where prefect.yaml lives. - This fixes the FileNotFoundError you’re seeing, but it ties your config to that host’s filesystem layout. If you run deploy from another machine (CI, laptop), those absolute paths must also exist there or deploy will fail. If you go this route:

deployments:
  - name: ingest-daily
    entrypoint: /opt/prefect/project/flows/ingest.py:ingest_flow
    work_pool:
      name: rhel-process-pool
      job_variables:
        working_dir: /opt/prefect/project
        stream_output: true

Caveats and alternatives - Portability: Absolute paths are brittle across machines. If you need to deploy from elsewhere, prefer: - Module-path entrypoints (install your code in the Conda env and use acme_pipelines.flows.ingest:ingest_flow), or - Git pull steps + set_working_directory so deploy doesn’t depend on local files. - If you cannot make files visible at deploy time, you can also add enforce_parameter_schema: false to skip flow import during deploy, but you still need the files present at runtime. Want me to patch your current prefect.yaml to absolute paths? If you share your code root (e.g., /opt/prefect/project) and one failing entrypoint, I’ll provide the exact edits.

Thanks for the logs. In Prefect 3.x, a flow run in state “Crashed” with “Process for flow run … exited with status code: 1” means the subprocess died before your flow code started. The “Running set_working_directory step…” line is part of the pull steps that happen just before the subprocess is launched, so the failure is almost always one of: - Invalid working directory or entrypoint path - Missing imports/dependencies in the subprocess environment - The worker isn’t running from the intended Conda environment - Storage/pull steps didn’t put the repo/files where the subprocess expects Quick checklist to isolate the cause: 1) Verify the entrypoint and files exist where the worker runs - Entrypoint must be

./path/to/file.py:flow_function_name

relative to your project root at deployment time, and the same path must exist on the worker host at runtime. - Inspect what Prefect recorded:

prefect deployment inspect <flow-name>/<deployment-name>

Look for “entrypoint”, “pull steps”, and any working_dir shown. 2) Confirm the working directory used by the process worker - If you rely on local files, set the work pool’s Process “base job template” working_dir to the folder on the host that contains your flows and data. - Ensure the worker user can cd into that directory on RHEL9 (permissions/SELinux). Docs: Workers and Process worker API 3) Make sure the worker is started from your shared Conda env - Since you have 50–100 flows sharing one environment, the simplest pattern is: - Activate your Conda env - Start the worker from that env This ensures the subprocess inherits the correct Python and dependencies. - Quick sanity check on the worker host, in the same shell you’ll use to start the worker:

which python
  python -c "import prefect; import your_dependency"

If that fails, the flow subprocess will likely exit 1. 4) Get the actual Python error by executing the flow run locally - On the worker machine, from the same env and directory you expect at runtime:

prefect flow-run execute <flow_run_id>

This prints the traceback that the generic exit code 1 is hiding. If you can’t reproduce, it’s usually a working_dir or path discrepancy. 5) Turn on more logging while you test - Start the worker with DEBUG logs to capture additional hints:

export PREFECT_LOGGING_LEVEL=DEBUG
  prefect worker start -p <your-work-pool>

6) Minimal test to validate cwd and sys.path - Create a tiny flow in the same project and deploy it:

from prefect import flow
  import os, sys

  @flow
  def debug_env():
      print("CWD:", os.getcwd())
      print("Files:", os.listdir(os.getcwd()))
      print("Python:", sys.executable)
      print("sys.path:", sys.path)

  if __name__ == "__main__":
      debug_env()

If this runs, your working_dir/env are correct; if not, it’ll reveal path/env issues. Notes specific to your setup (RHEL9 + local paths + shared Conda env): - Keep storage “None/local” so Prefect uses your existing files on the host. If you use git pull steps, confirm the clone destination and that your entrypoint path is valid in that directory after the pull. - Linux paths are case-sensitive; entrypoint must match the actual filename and function name exactly. - If SELinux is enforcing, confirm the worker user has permissions to read and cd into your project directories. Could you share a few details so we can pinpoint this fast? - The deployment’s entrypoint and pull steps from

prefect deployment inspect

- How you start the worker (the exact command) and which Conda env it’s running in - Whether you’ve set a working_dir in the work pool’s Process base job template (and what it’s set to) - The output of running

prefect flow-run execute <the failing run id>

on the worker host Helpful references:

- Deployments - Crashed state - Process worker config - on_crashed hook

Yes—this pinpoints the issue. Your flow subprocess is importing a third‑party “pathlib” backport from site‑packages instead of the standard library module. That backport tries to do

from collections import Sequence

, which is incompatible with Python 3.10+ (moved to

collections.abc

). On Python 3.13 it raises: ImportError: cannot import name 'Sequence' from 'collections' You can see it clearly in the path: .../site-packages/pathlib.py instead of the stdlib: .../lib/python3.13/pathlib.py What to do - Remove the backport from the environment the worker uses:

# in the same env your worker uses
  python -V
  python -c "import pathlib; print(pathlib.__file__)"
  pip show pathlib
  pip uninstall -y pathlib
  # verify stdlib is now used
  python -c "import pathlib; print(pathlib.__file__)"

The final print should point to something like .../lib/python3.13/pathlib.py, not site-packages. - If you can’t uninstall because something else depends on it, ensure the worker runs from a clean env without that package or set PYTHONNOUSERSITE=1 for the worker process to ignore user site-packages:

export PYTHONNOUSERSITE=1
  prefect worker start -p <your-work-pool>

- Make sure you start the worker from your intended Conda env (activate it first), since all flows share that env:

conda activate <your-env>
  which python
  python -V
  python -c "import pathlib; print(pathlib.__file__)"
  prefect worker start -p <your-work-pool>

Why the crash looked like this - The process worker launches a subprocess that imports Prefect; Prefect imports

pathlib

. Because the backport shadows the stdlib, the import fails immediately and the subprocess exits with status 1, putting the flow run into “Crashed”. FYI: Prefect 3.x supports Python 3.13, so once the stdlib

pathlib

is used, this specific error should go away. See the release notes highlighting Python 3.13 support: Prefect 3.3 release notes. If you remove the backport and still see issues, please share: - Output of

python -c "import pathlib; print(pathlib.__file__)"

from the worker env - How you’re starting the worker - The result of

prefect flow-run execute <failing-run-id>

after the change We’ll get you unblocked quickly.

Yes—this usually comes down to the Python environment the worker uses at runtime being different from the one you used locally. Key behavior in Prefect 3.x with process workers: - The subprocess that executes your flow uses the same Python interpreter and site-packages as the worker process. - If you start the worker from an environment that includes a third‑party

pathlib

backport, that is the

pathlib

your flow will import—even if your local dev shell uses a different env where that backport isn’t present. Why it worked locally but failed in deployment: - Locally: you likely ran with a different Python/Conda env that didn’t have the

pathlib

backport, or used an older Python where that backport didn’t explode. - Worker: it’s starting from an env that has

site-packages/pathlib.py

on

sys.path

(your log shows it under

/ctc/bundles/maestro/.../site-packages/pathlib.py

), which shadows the stdlib module and fails on Python 3.13. How to align the environments and fix it: 1) Start the worker from your intended shared Conda env - Activate the env, then start the worker from that shell:

conda activate <your-env>
  which python
  python -V
  python -c "import pathlib; print(pathlib.__file__)"
  prefect worker start -p <your-work-pool>

The

pathlib.__file__

must point to something like

.../lib/python3.13/pathlib.py

(stdlib), not

.../site-packages/pathlib.py

. 2) Remove or block the third‑party pathlib backport from the worker env - If it’s installed in that env:

pip show pathlib
  pip uninstall -y pathlib
  python -c "import pathlib; print(pathlib.__file__)"

- If it’s coming from a user site directory, disable user site-packages for the worker:

export PYTHONNOUSERSITE=1
  prefect worker start -p <your-work-pool>

3) If running the worker under systemd or another service manager, explicitly activate the Conda env - Example systemd service snippet:

[Service]
  Environment=PYTHONNOUSERSITE=1
  ExecStart=/bin/bash -lc 'source /etc/profile.d/conda.sh && conda activate <your-env> && prefect worker start -p <your-work-pool>'
  WorkingDirectory=/path/to/your/project

4) Verify at runtime - Deploy a tiny debug flow to print the interpreter and the imported pathlib path:

from prefect import flow
  import sys, pathlib
  @flow
  def debug_env():
      print("Python:", sys.executable)
      print("pathlib:", pathlib.__file__)
  if __name__ == "__main__":
      debug_env()

If this prints stdlib

pathlib

, you’re good. References: - Workers - Process worker API - Crashed state If you can share how you’re starting the worker (service vs shell) and the output of:

python -V
python -c "import pathlib; print(pathlib.__file__)"

from that same context, I can suggest the most direct change for your setup.