https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm having some trouble with my prefe...
# ask-marvin
m
@Marvin I'm having some trouble with my prefect worker logs. In my python code I set the default log level to INFO using: 
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
but it seems like prefect is overwriting this, because I'm only seeing warning-level (and higher) logs from my flow runs. And I'd also like apply structured (json) formatting to the logs. Can you tell me how to configure my work to also log info-level logs lines and to format logs as structured json log lines
m
✅ thought for 136.3 seconds
Yes — 
logging.basicConfig(...)
is usually not the right way to configure logging with Prefect. Prefect sets up its own logging config when it is imported, so your 
basicConfig(level=<http://logging.INFO|logging.INFO>)
often gets overridden. That’s why you’re only seeing `WARNING`+ in worker/stdout logs. What to do instead For Prefect 3.x, configure logging with Prefect settings / logging config, not 
basicConfig
. Why you only see warnings There are really two different log paths involved: 1. Flow/task run logs These are emitted by 
get_run_logger()
and sent to the Prefect API/UI. 2. Worker / console logs These go through the root/console logger, which defaults to 
WARNING
. So it’s very common for: - logs to exist in the Prefect UI - but only `WARNING`+ to show up in the worker process output If you want 
INFO
in worker logs/stdout, you need to lower the root logger level. Enable INFO-level logs in worker output Set:
Copy code
bash
export PREFECT_LOGGING_ROOT_LEVEL=INFO
You can also persist it with Prefect config:
Copy code
bash
prefect config set PREFECT_LOGGING_ROOT_LEVEL=INFO
That’s the key setting for seeing 
INFO
lines in worker/console output. If you also want Prefect’s own loggers at INFO, that is already the default, but you can be explicit:
Copy code
bash
export PREFECT_LOGGING_LEVEL=INFO
Important: set env vars before the worker starts. Example:
Copy code
bash
export PREFECT_LOGGING_ROOT_LEVEL=INFO
export PREFECT_LOGGING_LEVEL=INFO
prefect worker start --pool my-pool
Structured JSON logs For JSON logs, the most reliable approach is to provide a custom Prefect logging config file and point Prefect at it. Create a file like `~/.prefect/logging.yml`:
Copy code
yaml
version: 1
disable_existing_loggers: false

formatters:
  json:
    class: prefect.logging.formatters.JsonFormatter
    format: default

handlers:
  console:
    class: prefect.logging.handlers.PrefectConsoleHandler
    level: 0
    formatter: json
    stream: <ext://sys.stderr>

  api:
    class: prefect.logging.handlers.APILogHandler
    level: 0

  worker_api:
    class: prefect.logging.handlers.WorkerAPILogHandler
    level: 0

loggers:
  prefect:
    level: INFO
    handlers: [console]
    propagate: false

  prefect.flow_runs:
    level: NOTSET
    handlers: [api]
    propagate: false

  prefect.task_runs:
    level: NOTSET
    handlers: [api]
    propagate: false

  prefect.workers:
    level: NOTSET
    handlers: [worker_api, console]
    propagate: false

root:
  level: INFO
  handlers: [console]
Then point Prefect at it:
Copy code
bash
export PREFECT_LOGGING_SETTINGS_PATH=~/.prefect/logging.yml
Then start your worker. What this does - sets root logger to 
INFO
- formats console output as JSON - keeps flow/task logs going to the API - also sends worker logs to console as JSON Notes on your application logs If you log from inside flows/tasks, prefer:
Copy code
python
from prefect import flow, task
from prefect.logging import get_run_logger

@task
def my_task():
    logger = get_run_logger()
    <http://logger.info|logger.info>("hello from task")

@flow
def my_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("hello from flow")
    my_task()
Using 
get_run_logger()
ensures logs are attached to the flow/task run and show up properly in Prefect. If you use plain 
logging.getLogger(__name__)
, those logs may only go to stdout/stderr unless you explicitly configure those loggers too. If you want third-party library logs too You can include extra loggers with:
Copy code
bash
export PREFECT_LOGGING_EXTRA_LOGGERS=httpx,sqlalchemy
CLI reference I’d use:
Copy code
bash
prefect config set PREFECT_LOGGING_ROOT_LEVEL INFO
and for starting workers, the standard command is:
Copy code
bash
prefect worker start --pool my-pool
Recommended setup If your goal is “show INFO logs in worker logs and emit JSON lines”, I’d do this: 1. Create custom 
logging.yml
2. Set:
Copy code
bash
   export PREFECT_LOGGING_SETTINGS_PATH=~/.prefect/logging.yml
3. Start worker normally:
Copy code
bash
   prefect worker start --pool my-pool
Helpful links - Prefect logging customization docs: Logging customization - Prefect settings reference: Settings reference If you want, I can also give you: - a minimal JSON logging config just for workers, or - a config tailored for Kubernetes / Docker / CloudWatch / Datadog log collectors.
m
@Marvin thanks, I'm now getting more verbose logs in structured json format, by using the yml config. Currently I'm getting a lof of log in my console (stdout/stderr), but I'd love to also have these logs captured by prefect so I can see them in prefect UI. Is this possible without using custom prefect logger? (because I'd also like to capture logs from third-party libraries that use native python loggers).
m
✅ thought for 124.2 seconds
Yes — for Python logging logs, this is possible without rewriting everything to use 
get_run_logger()
. But there’s an important distinction: What Prefect can capture into the UI - Python 
logging
records from loggers that have Prefect’s API handler attached - 
print()
output if you enable 
log_prints=True
What Prefect does not automatically capture into the UI - arbitrary stdout/stderr text written outside Python logging - subprocess output unless you explicitly route it into logging/print So if a third-party library uses the standard 
logging
module, you can usually get those logs into the Prefect UI. Best option for third-party libraries: 
PREFECT_LOGGING_EXTRA_LOGGERS
 Prefect supports attaching its API log handler to additional logger names. Example:
Copy code
bash
export PREFECT_LOGGING_EXTRA_LOGGERS=requests,urllib3,sqlalchemy,dask
That tells Prefect to also capture those Python loggers and send them to the Prefect API/UI during flow/task runs. This is probably the cleanest solution for your case. If you use a custom 
logging.yml
 You can also explicitly wire those loggers to the 
api
handler in your config. Example:
Copy code
yaml
version: 1
disable_existing_loggers: false

formatters:
  json:
    class: prefect.logging.formatters.JsonFormatter
    format: default

handlers:
  console:
    class: prefect.logging.handlers.PrefectConsoleHandler
    level: 0
    formatter: json
    stream: <ext://sys.stderr>

  api:
    class: prefect.logging.handlers.APILogHandler
    level: 0

loggers:
  prefect:
    level: INFO
    handlers: [console]
    propagate: false

  prefect.flow_runs:
    level: NOTSET
    handlers: [api]
    propagate: false

  prefect.task_runs:
    level: NOTSET
    handlers: [api]
    propagate: false

  sqlalchemy:
    level: INFO
    handlers: [api, console]
    propagate: false

  urllib3:
    level: INFO
    handlers: [api, console]
    propagate: false

root:
  level: INFO
  handlers: [console]
That would send those libraries both to: - your JSON console logs - the Prefect UI Important limitation: only during a flow/task run context Prefect’s API log capture for run logs depends on flow/task run context. That means: - logs emitted inside a running flow/task can go to the UI - logs emitted by the worker process itself, outside the run context, generally won’t show up as flow run logs in the UI So if you want logs visible on the flow run page, they need to happen while the flow/task is executing. For 
print()
output If some code writes with 
print(...)
, you can capture that too:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def my_flow():
    print("this will go to Prefect logs")
or globally:
Copy code
bash
export PREFECT_LOGGING_LOG_PRINTS=true
But this only helps for 
print()
, not arbitrary 
sys.stdout.write(...)
or subprocess output. What will not be automatically captured - 
subprocess.run(...)
output to stdout/stderr - native extensions writing directly to stderr - shell commands unless you capture and re-log them - worker boot logs as run logs For those, you’d need to: - capture output in Python - emit it through logging or 
print()
Recommended setup for your case Since you already have structured JSON console logs, I’d suggest: 1. Keep your custom 
logging.yml
2. Add the third-party loggers you care about to either: - 
PREFECT_LOGGING_EXTRA_LOGGERS
, or - explicit 
loggers:
entries with 
handlers: [api, console]
3. Enable 
log_prints=True
if relevant Example env var approach:
Copy code
bash
export PREFECT_LOGGING_ROOT_LEVEL=INFO
export PREFECT_LOGGING_EXTRA_LOGGERS=sqlalchemy,urllib3,requests
export PREFECT_LOGGING_LOG_PRINTS=true
export PREFECT_LOGGING_SETTINGS_PATH=~/.prefect/logging.yml
One gotcha: duplicate logs If a third-party logger propagates upward and you attach both 
api
and 
console
, you may see duplicates. If that happens, set:
Copy code
yaml
propagate: false
for those specific loggers.
Short answer - Yes for third-party libraries that use Python 
logging
- Yes for 
print()
if 
log_prints=True
- No, not automatically for arbitrary stdout/stderr text or subprocess output Docs: - Logging customization - Settings reference If you want, I can help you turn your current 
logging.yml
into a version that sends: - JSON to stdout/stderr, and - selected third-party loggers to the Prefect UI without duplicating lines.
4 Views
Open in Slack
PreviousNext
Powered by