Great question. Here are pragmatic best practices for managing multiple Prefect profiles (e.g., dev/prod) from CI in Prefect 3.x, plus copy‑pasteable snippets. Key ideas - Use profiles for non-secret, environment-specific settings (API URL, logging, timeouts). - Keep secrets out of profiles; inject them via your CI secret store (e.g.,

PREFECT_API_KEY

). - In CI, either: - Point at a repo-scoped

profiles.toml

and select the profile, or - Create/use an ephemeral profile per job, or - Skip profiles entirely and set environment variables (simplest). - Validate and print effective config in CI for easy debugging. What controls profiles - Environment variables: -

PREFECT_PROFILES_PATH

— path to a profiles file (TOML) -

PREFECT_PROFILE

— selects the active profile by name -

PREFECT_API_URL

,

PREFECT_API_KEY

— commonly set per environment - Default location: - If you don’t set

PREFECT_PROFILES_PATH

, Prefect loads from

~/.prefect/profiles.toml

under

PREFECT_HOME

(defaults to

~/.prefect

). - CLI (non-interactive by default): -

prefect profile ls|create|use|inspect

-

prefect config set|unset|view|validate

Patterns that work well in CI 1) Repo-scoped profiles (recommended when you want consistent, reviewed config) - Commit a non-secret profiles file, then select the target profile in CI. - Example repo file:

# .ci/profiles.toml
  [dev]
  PREFECT_API_URL = "<http://localhost:4200/api>"
  PREFECT_LOGGING_LEVEL = "DEBUG"

  [prod]
  PREFECT_API_URL = "<https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>>"
  PREFECT_LOGGING_LEVEL = "INFO"

- Example CI steps:

# environment
  export PREFECT_PROFILES_PATH="$GITHUB_WORKSPACE/.ci/profiles.toml"
  export PREFECT_PROFILE="prod"
  export PREFECT_API_KEY="${{ secrets.PREFECT_API_KEY }}"  # keep secrets in CI

  # sanity check the active profile and resolved settings
  prefect profile use "$PREFECT_PROFILE"
  prefect profile inspect           # shows values from the selected profile
  prefect config validate
  prefect config view               # shows final settings after env overrides

  # deploy / run commands
  prefect deploy ./flows/flow.py:my_flow --name my-deploy

Notes: - Env vars override profile values, so secrets injected via CI take precedence. - Works well with matrix builds (set

PREFECT_PROFILE=dev|prod

per job). 2) Ephemeral per-job profile (no file in repo, fully driven by CI) - Use a temporary profile:

export PREFECT_API_KEY="${{ secrets.PREFECT_API_KEY }}"
  prefect profile create ci
  prefect profile use ci
  prefect config set PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/<acct>/workspaces/<ws>>"
  prefect config set PREFECT_LOGGING_LEVEL="INFO"

  prefect config validate
  prefect deploy ./flows/flow.py:my_flow --name my-deploy

Notes: - Writes to the runner’s ephemeral `~/.prefect/profiles.toml`; nothing persists after the job. - Keep secrets as env vars when possible to avoid writing them to disk. 3) Environment-variables only (simplest and fully ephemeral) - Skip profiles; just export what you need:

export PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/<acct>/workspaces/<ws>>"
  export PREFECT_API_KEY="${{ secrets.PREFECT_API_KEY }}"
  export PREFECT_LOGGING_LEVEL="INFO"

  prefect config validate
  prefect deploy ./flows/flow.py:my_flow --name my-deploy

Notes: - Great for minimalism; no profile switching logic required. - Use when you only need a handful of settings per environment. Secrets handling - Do not commit secrets to

profiles.toml

. - Prefer CI secret stores to inject

PREFECT_API_KEY

and any integration credentials. - Optionally use Prefect Cloud variables/blocks for runtime secrets retrieval by flows; CI only needs the

PREFECT_API_KEY

to deploy. Helpful CLI you can use in CI - List and select profiles: ``` prefect profile ls prefect profile use prod

prefect profile inspect # or: prefect profile inspect prod

- Set/unset settings in the active profile:

prefect config set PREFECT_API_URL="..." prefect config unset PREFECT_LOGGING_LEVEL

- Validate and view the resolved configuration:

prefect config validate prefect config view ``` Links - Concepts: Settings and profiles (Prefect 3.x) - API reference: Settings/profiles API - Security notes: Security and settings If you share which CI system you’re using (GitHub Actions, GitLab, CircleCI, etc.), I can drop in a ready-to-use job snippet tailored to it.

It looks like Prefect isn’t seeing your profile because the settings are in the wrong file. In Prefect 3.x: - Profiles live in a TOML file called

profiles.toml

(not

prefect.toml

) - By default, Prefect reads

~/.prefect/profiles.toml

- You can point to a repo-scoped file with

PREFECT_PROFILES_PATH

What to do 1) Move your profiles out of

prefect.toml

into a `profiles.toml`:

# ./profiles.toml (example)
[dev]
PREFECT_API_URL = "<http://127.0.0.1:4200/api>"
PREFECT_LOGGING_LEVEL = "DEBUG"

[prod]
PREFECT_API_URL = "<https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>>"
PREFECT_LOGGING_LEVEL = "INFO"

2) Tell Prefect where that file is and select the profile:

export PREFECT_PROFILES_PATH="$PWD/profiles.toml"
prefect profile ls
prefect profile use prod
prefect profile inspect
prefect config view          # show resolved config; should include PREFECT_API_URL

3) If you need to keep

prefect.toml

for defaults, that’s fine—but it cannot define multiple named profiles. Use

profiles.toml

for `[dev]`/`[prod]` sections. Quick alternative (no file changes)

prefect profile create prod
prefect profile use prod
prefect config set PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/<acct>/workspaces/<ws>>"
prefect config view

If you still see the same error after this, please share: - The path you placed

profiles.toml

in - The exact contents of your

[prod]

section (feel free to redact IDs) - Output of:

prefect profile ls
prefect profile inspect prod
prefect config view

Links: - Settings and profiles: Prefect 3.x docs