<@ULVA73B9P> my `PREFECT_LOGGING_SETTINGS_PATH` ennvar is not being respected and not being loaded a...
l
@Marvin my
PREFECT_LOGGING_SETTINGS_PATH
ennvar is not being respected and not being loaded any idea why? this is in GKE
Copy code
appuser@nifty-rat-h2z6h-6w6kn:~/workspace$ python - <<'PY'
from prefect.settings import PREFECT_LOGGING_SETTINGS_PATH
from prefect.logging.configuration import load_logging_config
from pathlib import Path
p = Path(PREFECT_LOGGING_SETTINGS_PATH.value() or "")
print("Prefect sees path:", p)
print("Exists:", p.exists())
if p.exists():
    cfg = load_logging_config(p)
    print("Formatter class for json:",
          cfg.get("formatters", {}).get("json", {}).get("()", "<missing>"))
PY
Prefect sees path: /opt/prefect/logging.yml
Exists: False
appuser@nifty-rat-h2z6h-6w6kn:~/workspace$ echo $PREFECT_LOGGING_SETTINGS_PATH
/app/config/logging/gcp.yml
m
thought for 540.5 seconds
Short answer: you’re setting the wrong env var. In Prefect 3.x, the knob to point at a custom logging YAML is the nested setting
logging.config_path
, which maps to the environment variable
PREFECT_LOGGING__CONFIG_PATH
(note the double underscore for nested settings). The
PREFECT_LOGGING_SETTINGS_PATH
object you’re printing is the default/internal path used by Prefect’s base images (
/opt/prefect/logging.yml
) and isn’t the user-facing override. Why your snippet shows
/opt/prefect/logging.yml
-
PREFECT_LOGGING_SETTINGS_PATH.value()
returns the package default path, not the nested logging setting. - Your shell has
PREFECT_LOGGING_SETTINGS_PATH=/app/config/logging/gcp.yml
, but Prefect doesn’t read that to override logging in 3.x. What to do instead 1) Set the correct env var and make sure the file exists in the container: - Env var:
Copy code
export PREFECT_LOGGING__CONFIG_PATH=/app/config/logging/gcp.yml
- Verify in Python:
Copy code
from prefect.settings import get_current_settings
s = get_current_settings()
print("Prefect config_path:", s.logging.config_path)
- Confirm the file exists:
Copy code
ls -l /app/config/logging/gcp.yml
2) If you’re on GKE/Kubernetes: - The env var must be present in the flow-run pod (not just the worker process). Add it to your Kubernetes work pool base job template or deployment’s job variables so it’s injected into the flow-run container. - Also mount the YAML into the flow-run container (via ConfigMap/volume or bake it into your image) at the same path you set above. Example snippet for a Kubernetes work pool base job template env entry:
Copy code
"env": [
  {"name": "PREFECT_LOGGING__CONFIG_PATH", "value": "/app/config/logging/gcp.yml"}
]
…and mount your ConfigMap/volume so
/app/config/logging/gcp.yml
exists. 3) Optional: set via profile instead of env - In a
prefect.toml
baked into your image:
Copy code
[logging]
config_path = "/app/config/logging/gcp.yml"
- Ensure the profile is active in the container (e.g.,
PREFECT_PROFILE
set appropriately). Quick sanity check you can run now
Copy code
python - <<'PY'
import os
from pathlib import Path
from prefect.settings import get_current_settings

print("PREFECT_LOGGING__CONFIG_PATH:", os.getenv("PREFECT_LOGGING__CONFIG_PATH"))
s = get_current_settings()
print("settings.logging.config_path:", s.logging.config_path)
p = Path(s.logging.config_path or "")
print("File exists:", p.exists(), "->", p)
PY
Docs for reference: - Logging customization: Customize logging - Settings and profiles: Settings and profiles If you share how you’re injecting env and mounting the file in your GKE setup (work pool template vs deployment job variables), I can suggest the exact configuration snippet.
l
@Marvin directly from the docs. please provide a github link to back up your claims
Specify a custom path to your
logging.yml
file using the
PREFECT_LOGGING_SETTINGS_PATH
setting.
m
thought for 209.1 seconds
Thanks for calling that out — can you share the exact docs URL where that sentence appears? I can’t find that wording in the current 3.x docs repo and I want to make sure we’re looking at the same source. What the code shows today (Prefect 3.x): - The user-facing setting to point at a logging YAML is
logging.config_path
(a Path field on LoggingSettings) - Source: src/prefect/settings/models/logging.py - The base image default path is
/opt/prefect/logging.yml
, exposed as
DEFAULT_LOGGING_SETTINGS_PATH
- Source: src/prefect/logging/configuration.py About the environment variable name - Prefect settings are nested; environment variables target nested fields using a delimiter. In practice, this maps
logging.config_path
to
PREFECT_LOGGING__CONFIG_PATH
(double underscore). - You can verify at runtime:
Copy code
from prefect.settings import get_current_settings
print(get_current_settings().logging.config_path)
Why
PREFECT_LOGGING_SETTINGS_PATH
doesn’t work -
PREFECT_LOGGING_SETTINGS_PATH
is not the setting used to override the path in 3.x. It’s a default path constant used by the base image (hence why printing it shows
/opt/prefect/logging.yml
), but it is not the env var that the settings system reads to override logging. - To override, set
PREFECT_LOGGING__CONFIG_PATH=/app/config/logging/gcp.yml
and ensure the file exists in the container. If you can link the docs page you quoted, I’ll open a PR to correct it (or reconcile if there’s a nuance I’m missing). In the meantime, the GitHub sources above reflect the behavior you’re bumping into.
l
@Marvin okay i got it to work thanks why is the prefect dashboard showing some of my logs as structuredoutput? {"name":"prefect.task_runs","msg":"Finished in state Completed()","args":[],"levelname":"INFO","levelno":20,"pathname":"/app/venv/lib/python3.12/site-packages/prefect/task_engine.py","filename":"task_engine.py","module":"task_engine","exc_info":null,"exc_text":null,"stack_info":null,"lineno":286,"funcName":"log_finished_message","created":1757466434.937238,"msecs":937.0,"relativeCreated":32549.89743232727,"thread":138243989207936,"threadName":"MainThread","processName":"MainProcess","process":19,"taskName":null,"task_run_id":"01993129-8efb-71de-9f2b-fc3f94c967ff","flow_run_id":"4a1b474d-2b35-4d01-8a8b-35f54299bd06","task_run_name":"iwp_task-eb6","task_name":"iwp_task","flow_run_name":"garnet-worm","flow_name":"dev-pipeline-single-flow","severity":"INFO","message":"Finished in state Completed()"}
m
thought for 123.3 seconds
Great to hear it’s working now! About “structuredoutput” in the UI: that’s happening because the logs hitting the Prefect API are JSON-formatted (structured) instead of plain text. The Prefect UI detects JSON payloads and renders them as structured output. In your snippet, the message is a full JSON object, which is why the dashboard annotates it that way. Why this happens - If your logging.yml applies a JSON formatter to the handler that sends logs to the Prefect API (APILogHandler), the API receives a JSON string. The UI then treats it as structured output. - This often happens when you want JSON logs for GKE/GCP (to integrate with Cloud Logging) and you attach the same JSON formatter to all handlers, including the API handler. How to keep JSON for your container logs but plain text in the Prefect UI - Use JSON formatting only on your console/stream handler (for GKE/GCP ingestion). - Keep the APILogHandler’s formatter plain (or use Prefect’s default formatter) so the UI gets a simple message string. Example logging.yml layout
Copy code
version: 1
formatters:
  plain:
    format: "%(asctime)s | %(levelname)s | %(name)s - %(message)s"
    datefmt: "%Y-%m-%d %H:%M:%S"
    class: logging.Formatter

  json:
    "()": prefect.logging.formatters.JsonFormatter
    fmt: default  # or 'pretty'

handlers:
  console:
    class: prefect.logging.handlers.PrefectConsoleHandler
    level: INFO
    formatter: json           # keep JSON for stdout/stderr

  api:
    "()": prefect.logging.handlers.APILogHandler
    level: INFO
    formatter: plain          # IMPORTANT: plain for the API/UI

loggers:
  prefect:
    level: INFO
    handlers: [console, api]
    propagate: False

root:
  level: INFO
  handlers: [console, api]
Key classes in Prefect you’re touching - APILogHandler (sends logs to the Prefect API): handlers.py - JsonFormatter (JSON output): formatters.py Docs - Logging customization: Customize logging If you share your current handler/formatter blocks, I can suggest the exact edits to split “JSON to stdout” from “plain to Prefect API.”