Gio
09/04/2025, 10:06 PMMarvin
09/04/2025, 10:06 PMMarvin
09/04/2025, 10:11 PMlogging.yml
at PREFECT_HOME/logging.yml
in the runtime environment, or point to it with a setting (see docs below).
- Example logging.yml (dictConfig):
version: 1
disable_existing_loggers: false
formatters:
standard:
format: "%(asctime)s | %(levelname)s | %(name)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
formatter: standard
level: INFO
stream: <ext://sys.stdout>
loggers:
prefect:
level: INFO
handlers: [console]
propagate: false
root:
level: INFO
handlers: [console]
- Set Prefect logging settings via environment variables
- You can set things like:
- turning on log forwarding to the API
- enabling log_prints
- adding extra_loggers
- colors/markup, etc.
- The exact setting names are documented in the Settings reference (linked below). Add them to the work pool’s env
so every flow run from that pool inherits them.
How to apply this to a work pool
- Edit the work pool’s base job template (UI or CLI) and add your env vars under variables.env
.
- CLI example (Process worker shown, but works similarly for Docker/Kubernetes):
prefect work-pool get-default-base-job-template --type process > process.json
# Edit process.json and add your env vars, for example:
# {
# "job_configuration": { ... },
# "variables": {
# "env": {
# "PREFECT_LOGGING_LOG_PRINTS": "true",
# "PREFECT_LOGGING_EXTRA_LOGGERS": "urllib3,boto3",
# "PREFECT_LOGGING_SETTINGS_PATH": "/etc/prefect/logging.yml"
# }
# }
# }
prefect work-pool update my-pool --base-job-template process.json
- For deployments, you can also set env per-deployment via job_variables.env
in your `prefect.yaml`:
deployments:
- name: etl
work_pool:
name: my-pool
job_variables:
env:
PREFECT_LOGGING_SETTINGS_PATH: /etc/prefect/logging.yml
PREFECT_LOGGING_LOG_PRINTS: "true"
Notes
- If you also want to change the worker process’s own logs (not just flow runs), set the same env vars in the environment that starts the worker (e.g., systemd service, container env for the worker).
- Kubernetes/Docker workers will pass variables.env
into the pod/container; Process workers set them directly in the subprocess that runs the flow.
Docs and references
- Logging customization (how to use a logging.yml and available knobs): Logging customization
- Settings and profiles (env vars and profiles): Settings and profiles
- Managing work pools and job templates: Manage work pools
- Customize job variables for deployments: Customize job variables
Quick check: what worker type are you using (Process, Docker, or Kubernetes), and do you primarily want to change level/format, forward logs, or include additional libraries’ logs? I can share a tailored template snippet.