https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I set all the settings in .env file. ...
# ask-marvin
v
@Marvin I set all the settings in .env file. When I serve the flow, prefect correctly reads .env, connects to the server and deploys the flow as "served". But when I click to run that flow, I get an error
Copy code
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/engine.py", line 100, in <module>
    with handle_engine_signals(flow_run_id):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/engine.py", line 77, in handle_engine_signals
    engine_logger.error(msg, exc_info=True)
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1568, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1684, in _log
    self.handle(record)
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1695, in handle
    maybe_record = self.filter(record)
                   ^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 858, in filter
    result = f.filter(record)
             ^^^^^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/logging/filters.py", line 38, in filter
    if PREFECT_API_KEY:
       ^^^^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/settings/legacy.py", line 71, in __bool__
    return bool(self.value())
                ^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/settings/legacy.py", line 59, in value
    return self.value_from(get_current_settings())
                           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/settings/context.py", line 15, in get_current_settings
    from prefect.context import SettingsContext
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/context.py", line 950, in <module>
    GLOBAL_SETTINGS_CONTEXT: SettingsContext = root_settings_context()
                                               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/context.py", line 909, in root_settings_context
    profiles = prefect.settings.load_profiles()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/settings/profiles.py", line 307, in load_profiles
    current_settings = get_current_settings()
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/prefect/settings/context.py", line 21, in get_current_settings
    return Settings()
           ^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 188, in __init__
    super().__init__(
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 189, in __init__
    **__pydantic_self__._settings_build_values(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 424, in _settings_build_values
    source_state = source()
                   ^^^^^^^^
  File "/Users/somename/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/sources/base.py", line 492, in __call__
    raise SettingsError(
pydantic_settings.exceptions.SettingsError: error parsing value for field "custom_headers" from source "EnvFilterSettingsSource"
Looks like run creates a subprocess in my local machine, but can't find the .env file to read it. How can I solve without sourcing .env to the system?
m
✅ thought for 744.8 seconds
Thanks for the detailed traceback — that actually points to the real culprit. Key point: the run isn’t failing because it can’t find your .env. It’s failing because the value of your custom headers setting cannot be parsed. The error comes from pydantic_settings.exceptions.SettingsError: error parsing value for field "custom_headers" from source "EnvFilterSettingsSource" That means 
PREFECT_CLIENT_CUSTOM_HEADERS
is present (from your environment or .env), but its value is not in a format Prefect can parse. What to do 1) Fix the headers format - Easiest, robust option: set it as a JSON object string. - Examples you can put in your .env:
Copy code
PREFECT_CLIENT_CUSTOM_HEADERS='{"Authorization":"Bearer <token>"}'
or multiple headers
Copy code
PREFECT_CLIENT_CUSTOM_HEADERS='{"X-Org":"my-org","X-Env":"dev"}'
- If you prefer a delimited string, make sure it’s strictly key:value pairs separated by commas (no trailing commas, watch spaces/quotes):
Copy code
PREFECT_CLIENT_CUSTOM_HEADERS='X-Org:my-org,X-Env:dev'
If in doubt, use JSON; Pydantic will reliably parse dicts from JSON strings. 2) Verify your settings load the way you expect - After updating .env, run:
Copy code
prefect config view --show-sources
You should see 
client.custom_headers
and where it’s coming from. If it still errors, temporarily remove 
PREFECT_CLIENT_CUSTOM_HEADERS
from your .env to confirm the run works, then add it back with corrected formatting. 3) If you want to avoid relying on .env resolution in subprocesses When you click “Run”, Prefect starts a subprocess. While environment variables are inherited, .env discovery relies on the current working directory. If you don’t want to “source” system-wide: - Load .env programmatically in your flow entry script (per-process only):
Copy code
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv(), override=False)
- Or use a Prefect profile instead of .env so all subprocesses see the same settings:
Copy code
prefect profile create local
prefect profile use local
prefect config set PREFECT_API_URL="<your api url>"
prefect config set PREFECT_API_KEY="<your api key>"
prefect config set PREFECT_CLIENT_CUSTOM_HEADERS='{"X-Org":"my-org"}'
Helpful references - Settings and profiles: Settings and profiles - Settings reference: Settings API reference - Related discussion about custom headers: Passing custom headers to API calls If you can share (sanitized) what you currently have in 
PREFECT_CLIENT_CUSTOM_HEADERS
in your .env, I can spot-correct the formatting for you.
v
@Marvin
Copy code
$uv run prefect config view --show-sources
Bytecode compiled 7940 files in 595ms
🚀 you are connected to:
<https://prefect-server.tech>
PREFECT_PROFILE='local'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='<https://prefect-server.tech/api>' (from .env file)
PREFECT_CLIENT_CUSTOM_HEADERS='{"Authorization": "Bearer eyJ0eXAiOiJKV1...kPTEgCQYKmiFOFA"}' (from .env file)
and when I serve and run, I get that:
Copy code
$ uv run orchestrator serve -d InheritBarDev
Bytecode compiled 7940 files in 241ms
Your flow 'InheritBaseDev' is being served and polling for scheduled runs!

To trigger a run for this flow, use the following command:

        $ prefect deployment run 'InheritBaseDev/InheritBarDev'

You can also run your flow via the Prefect UI: <https://prefect-server-pricing-pi-dev-public.ew1d1.k8s-dev.flix.tech/deployments/deployment/212d79f5-5eb2-4358-a090-f4420b158d2c>

13:34:02.657 | INFO    | prefect.flow_runs.runner - Runner 'InheritBarDev' submitting flow run '0cfd8a2e-8c4a-427e-b7b3-3576a4ca2b09'
13:34:02.775 | INFO    | prefect.flow_runs.runner - Opening process...
13:34:02.797 | INFO    | prefect.flow_runs.runner - Completed submission of flow run '0cfd8a2e-8c4a-427e-b7b3-3576a4ca2b09'
Traceback (most recent call last):
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/sources/base.py", line 495, in __call__
    field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/sources/providers/env.py", line 120, in prepare_field_value
    raise e
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/sources/providers/env.py", line 117, in prepare_field_value
    value = self.decode_complex_value(field_name, field, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/sources/base.py", line 186, in decode_complex_value
    return json.loads(value)
           ^^^^^^^^^^^^^^^^^
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/json/decoder.py", line 338, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/json/decoder.py", line 354, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/engine.py", line 57, in handle_engine_signals
    yield
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/engine.py", line 101, in <module>
    from prefect.flow_engine import (
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/flow_engine.py", line 33, in <module>
    from prefect import Task
  File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/__init__.py", line 120, in __getattr__
    module = importlib.import_module(mname, package=package)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/main.py", line 6, in <module>
    from prefect.transactions import Transaction
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/transactions.py", line 27, in <module>
    from prefect.context import ContextModel
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/context.py", line 950, in <module>
    GLOBAL_SETTINGS_CONTEXT: SettingsContext = root_settings_context()
                                               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/context.py", line 909, in root_settings_context
    profiles = prefect.settings.load_profiles()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/settings/profiles.py", line 307, in load_profiles
    current_settings = get_current_settings()
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/prefect/settings/context.py", line 22, in get_current_settings
    return Settings()
           ^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 188, in __init__
    super().__init__(
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 189, in __init__
    **__pydantic_self__._settings_build_values(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 424, in _settings_build_values
    source_state = source()
                   ^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/sources/base.py", line 497, in __call__
    raise SettingsError(
pydantic_settings.exceptions.SettingsError: error parsing value for field "custom_headers" from source "EnvFilterSettingsSource"

...
13:34:03.209 | ERROR   | prefect.flow_runs.runner - Process for flow run 'wealthy-oryx' exited with status code: 1
13:34:03.435 | INFO    | prefect.flow_runs.runner - Reported flow run '0cfd8a2e-8c4a-427e-b7b3-3576a4ca2b09' as crashed: Flow run process exited with non-zero status code 1.
13:34:03.527 | INFO    | prefect.flow_runs.runner - Loading flow to check for on_crashed hooks
13:34:03.757 | INFO    | Flow run 'wealthy-oryx' - Downloading flow code from storage at '.'
If I print inside pydantic
Copy code
print(repr(field_value))
then during serve operation I get:
Copy code
'{"Authorization": "Bearer ey...
but during the run I get:
Copy code
"{'Authorization': 'Bearer ey
I wanted to find the bug and fix it in the source code(I will open PR, of course). But I struggle to find the source of the problem. Pls help.
m
Your message was too long, here's your message at the allowed limit:
Copy code
$uv run prefect config view --show-sources Bytecode compiled 7940 files in 595ms 🚀 you are connected to <https //prefect-server tech> PREFECT_PROFILE='local' PREFECT_API_KEY='****' (from profile) PREFECT_API_URL='<https //prefect-server tech/api>' (from env file) PREFECT_CLIENT_CUSTOM_HEADERS='{ Authorization Bearer eyJ0eXAiOiJKV1 kPTEgCQYKmiFOFA }' (from env file)
and when I serve and run I get that ```$ uv run orchestrator serve -d InheritBarDev Bytecode compiled 7940 files in 241ms Your flow 'InheritBaseDev' is being served and polling for scheduled runs! To trigger a run for this flow use the following command $ prefect deployment run 'InheritBaseDev/InheritBarDev' You can also run your flow via the Prefect UI <https //prefect-server-pricing-pi-dev-public ew1d1 k8s-dev flix tech/deployments/deployment/212d79f5-5eb2-4358-a090-f4420b158d2c> 13 34 02 657 | INFO | prefect flow_runs runner - Runner 'InheritBarDev' submitting flow run '0cfd8a2e-8c4a-427e-b7b3-3576a4ca2b09' 13 34 02 775 | INFO | prefect flow_runs runner - Opening process 13 34 02 797 | INFO | prefect flow_runs runner - Completed submission of flow run '0cfd8a2e-8c4a-427e-b7b3-3576a4ca2b09' Traceback (most recent call last) File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/pydantic_settings/sources/base py line 495 in call field_value = self prepare_field_value(field_name field field_value value_is_complex) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/pydantic_settings/sources/providers/env py line 120 in prepare_field_value raise e File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/pydantic_settings/sources/providers/env py line 117 in prepare_field_value value = self decode_complex_value(field_name field value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/pydantic_settings/sources/base py line 186 in decode_complex_value return json loads(value) ^^^^^^^^^^^^^^^^^ File /Users/vugar/ local/share/uv/python/cpython-3 12 10-macos-aarch64-none/lib/python3 12/json/__init__ py line 346 in loads return _default_decoder decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/ local/share/uv/python/cpython-3 12 10-macos-aarch64-none/lib/python3 12/json/decoder py line 338 in decode obj end = self raw_decode(s idx=_w(s 0) end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/ local/share/uv/python/cpython-3 12 10-macos-aarch64-none/lib/python3 12/json/decoder py line 354 in raw_decode obj end = self scan_once(s idx) ^^^^^^^^^^^^^^^^^^^^^^ json decoder JSONDecodeError Expecting property name enclosed in double quotes line 1 column 2 (char 1) The above exception was the direct cause of the following exception Traceback (most recent call last) File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/engine py line 57 in handle_engine_signals yield File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/engine py line 101 in <module> from prefect flow_engine import ( File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/flow_engine py line 33 in <module> from prefect import Task File <frozen importlib _bootstrap> line 1412 in _handle_fromlist File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/__init__ py line 120 in getattr module = importlib import_module(mname package=package) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/ local/share/uv/python/cpython-3 12 10-macos-aarch64-none/lib/python3 12/importlib/__init__ py line 90 in import_module return _bootstrap _gcd_import(name[level ] package level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/main py line 6 in <module> from prefect transacti
ons import Transaction File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/transactions py line 27 in <module> from prefect context import ContextModel File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/context py line 950 in <module> GLOBAL_SETTINGS_CONTEXT SettingsContext = root_settings_context() ^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/context py line 909 in root_settings_context profiles = prefect settings load_profiles() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/settings/profiles py line 307 in load_profiles current_settings = get_current_settings() ^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/prefect/settings/context py line 22 in get_current_settings return Settings() ^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/pydantic_settings/main py line 188 in init super() __init__( File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/pydantic/main py line 253 in init validated_self = self __pydantic_validator__ validate_python(data self_instance=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3 12/site-packages/pydantic_settings/main py line 189 in init **__pydantic_self__ _settings_build_values( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /Users/vugar/orchestrator/ venv/lib/python3
v
Copy code
$uv run prefect config view --show-sources
Bytecode compiled 7940 files in 595ms
🚀 you are connected to:
<https://prefect-server.tech>
PREFECT_PROFILE='local'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='<https://prefect-server.tech/api>' (from .env file)
PREFECT_CLIENT_CUSTOM_HEADERS='{"Authorization": "Bearer eyJ0eXAiOiJKV1...kPTEgCQYKmiFOFA"}' (from .env file)
and when I serve and run, I get that:
Copy code
$ uv run orchestrator serve -d InheritBarDev
Bytecode compiled 7940 files in 241ms
Your flow 'InheritBaseDev' is being served and polling for scheduled runs!

To trigger a run for this flow, use the following command:

        $ prefect deployment run 'InheritBaseDev/InheritBarDev'

You can also run your flow via the Prefect UI: <https://prefect-server-pricing-pi-dev-public.ew1d1.k8s-dev.flix.tech/deployments/deployment/212d79f5-5eb2-4358-a090-f4420b158d2c>

13:34:02.657 | INFO    | prefect.flow_runs.runner - Runner 'InheritBarDev' submitting flow run '0cfd8a2e-8c4a-427e-b7b3-3576a4ca2b09'
13:34:02.775 | INFO    | prefect.flow_runs.runner - Opening process...
13:34:02.797 | INFO    | prefect.flow_runs.runner - Completed submission of flow run '0cfd8a2e-8c4a-427e-b7b3-3576a4ca2b09'
Traceback (most recent call last):
  ...
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/json/decoder.py", line 338, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/.local/share/uv/python/cpython-3.12.10-macos-aarch64-none/lib/python3.12/json/decoder.py", line 354, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  ...
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 188, in __init__
    super().__init__(
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 189, in __init__
    **__pydantic_self__._settings_build_values(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 424, in _settings_build_values
    source_state = source()
                   ^^^^^^^^
  File "/Users/vugar/orchestrator/.venv/lib/python3.12/site-packages/pydantic_settings/sources/base.py", line 497, in __call__
    raise SettingsError(
pydantic_settings.exceptions.SettingsError: error parsing value for field "custom_headers" from source "EnvFilterSettingsSource"

...
13:34:03.209 | ERROR   | prefect.flow_runs.runner - Process for flow run 'wealthy-oryx' exited with status code: 1
13:34:03.435 | INFO    | prefect.flow_runs.runner - Reported flow run '0cfd8a2e-8c4a-427e-b7b3-3576a4ca2b09' as crashed: Flow run process exited with non-zero status code 1.
13:34:03.527 | INFO    | prefect.flow_runs.runner - Loading flow to check for on_crashed hooks
13:34:03.757 | INFO    | Flow run 'wealthy-oryx' - Downloading flow code from storage at '.'
If I print inside pydantic
Copy code
print(repr(field_value))
then during serve operation I get:
Copy code
'{"Authorization": "Bearer ey...
but during the run I get:
Copy code
"{'Authorization': 'Bearer ey
I wanted to find the bug and fix it in the source code(I will open PR, of course). But I struggle to find the source of the problem. Pls help @Marvin
m
✅ thought for 667.6 seconds
Thanks for the detailed traces — this explains the failure clearly. What’s happening - 
PREFECT_CLIENT_CUSTOM_HEADERS
must be valid JSON because pydantic_settings reads it from the environment and decodes it with 
json.loads
. - In your “serve” process, the value is a proper JSON string (double quotes) — you saw 
repr(field_value)
as 
'{"Authorization": "Bearer ..."}'
. - When the flow run is spawned in a subprocess, the environment variable is being overwritten to a Python dict string representation (single quotes), i.e. 
"{'Authorization': 'Bearer ..."}"
. This is not valid JSON, so pydantic_settings raises 
JSONDecodeError
and the flow run crashes. Where the bug likely lives - Prefect creates a new process for flow execution and injects the current settings into the child process environment. - Somewhere in this path, the settings values are serialized to environment strings using 
str(value)
instead of JSON for complex types — which turns dicts into single-quoted Python reprs. Files/places to look - Flow-run subprocess spawn: - src/prefect/flow_engine.py — see 
run_flow_in_subprocess
and any helpers it uses to compose the child env. - src/prefect/flow_runs/runner.py — this module logs “Opening process...”; it constructs the command/env and calls 
prefect.utilities.processutils.open_process
. - src/prefect/utilities/processutils.py — 
open_process
takes an 
env
kwarg that is passed to the subprocess; look for where that 
env
dict is built. - Settings export to env (the root cause): - src/prefect/settings/context.py - src/prefect/settings/base.py - src/prefect/settings/models/root.py - There’s a helper used by the runner/engine that turns the current Settings model into environment variables. In that code, dict/list/tuple/SecretStr values should be serialized to JSON or plain strings appropriately. Today it’s almost certainly doing 
str(value)
for mappings which produces single quotes. - Settings model that defines 
custom_headers
for type context: - src/prefect/settings/models/client.py — 
ClientSettings
includes 
custom_headers
(a mapping) and is parsed from env via pydantic_settings. The env source is EnvFilterSettingsSource which uses 
json.loads
for mappings. Proposed fix - In the function that converts Settings → env dict for the subprocess, ensure values are JSON-serialized when they are containers:
Copy code
import json
  from pydantic import SecretStr
  from enum import Enum
  from collections.abc import Mapping, Sequence

  def serialize_env_value(v):
      if isinstance(v, SecretStr):
          v = v.get_secret_value()
      if isinstance(v, (Mapping, Sequence)) and not isinstance(v, (str, bytes)):
          return json.dumps(v)
      if isinstance(v, bool):
          return "true" if v else "false"
      if isinstance(v, Enum):
          return v.value
      return str(v)

  # when building env:
  child_env[name] = serialize_env_value(value)
- Alternatively (more targeted), add a field serializer on 
ClientSettings.custom_headers
to ensure it’s dumped as JSON when exported to env. But since other settings may be mappings too, fixing the generic “settings to env” function is safer. Minimal repro you can use in tests 1) Set env: ``` export PREFECT_API_URL='http://localhost:4200/api'
export PREFECT_CLIENT_CUSTOM_HEADERS='{"Authorization":"Bearer foo"}'
Copy code
2) Run a trivial flow with subprocess execution and confirm the failure pre-fix, then success post-fix:
from prefect import flow from prefect.flow_engine import run_flow_in_subprocess @flow def f(): return "ok" if name == "__main__": # Should not crash after the fix p = run_flow_in_subprocess(f) p.join()
Copy code
Immediate workarounds
- Prefer setting the Authorization header via Prefect’s dedicated setting instead of `custom_headers`, which avoids JSON parsing entirely:
  - `PREFECT_API_AUTH_STRING='Bearer <token>'`
  - Then unset `PREFECT_CLIENT_CUSTOM_HEADERS`:
unset PREFECT_CLIENT_CUSTOM_HEADERS export PREFECT_API_AUTH_STRING='Bearer ey...'
Copy code
- The client will add the Authorization header automatically.

- If you must use custom headers for more than Authorization, consider setting them via code before making client calls (until the fix lands).

A couple of quick asks to help me validate:
- What Prefect version are you on? You can check with:
uv run python -c "import prefect, sys; print(prefect.version); print(sys.version)" ``` - If you’re up for a PR, I’m happy to review a patch that replaces 
str(value)
with the JSON-aware serializer as above in the settings→env export path. If you’d like, I can point to the exact function once you share the version/branch you’re targeting. References - Settings reference: Prefect 3 Settings - Client settings API: ClientSettings - Process utils: processutils
Open in Slack
PreviousNext
Powered by