<@ULVA73B9P> can the increased logging by flows sl...
# ask-marvin
r
@Marvin can the increased logging by flows slow down the flows? since the logs need to be sent over api to prefect server?
m
thought for 175.7 seconds
Short answer: Yes, very chatty logging can add some overhead — but in Prefect 3.x logs are shipped asynchronously and batched to minimize impact. Details: - Prefect’s APILogHandler/APILogWorker queue log records and send them to the API in the background. By default: - batch_interval ≈ 2s - batch_size ≈ 4 MB - max_log_size per record ≈ 1 MB - Because shipping is async and batched, normal INFO/DEBUG logging should have minimal effect on task/flow runtime. The main cases where you may notice impact: - Extremely high log volume (tight loops, large messages/objects) causing CPU/JSON-serialization overhead or queue pressure - Large log records (near the 1 MB cap) increasing serialization time - Network issues or API rate limits causing retries/backpressure (logs are retried with backoff) - Process shutdown: the log worker flushes outstanding logs, which can make exits feel “slow” if there’s a big backlog How to reduce overhead if needed: - Lower the log level in production: - Set
PREFECT_LOGGING_LEVEL=WARNING
- Disable shipping logs to the API (console-only logs): - Set
PREFECT_LOGGING_TO_API_ENABLED=false
- Tame volume: - Avoid logging inside tight loops or logging large payloads - Keep messages small; don’t log big dicts/dataframes - Adjust batching (trade latency for fewer API calls): -
PREFECT_LOGGING_TO_API_BATCH_INTERVAL=5
-
PREFECT_LOGGING_TO_API_BATCH_SIZE=<bytes>
- Keep
log_prints
off (it’s off by default) to avoid capturing every `print`: -
PREFECT_LOGGING_LOG_PRINTS=false
If you’re seeing slowdowns, can you share: - Approximate log rate (lines/second) and whether you’re logging in hot loops - Whether this is Prefect Cloud or self-hosted Server - Any large/structured payloads in messages References: - Logging handlers API: API: logging handlers - Logging customization: Advanced: logging customization - Settings and profiles: Concepts: settings and profiles - Rate limits: Concepts: rate limits