Jason Wiener
10/09/2025, 7:31 PMMarvin
10/09/2025, 7:31 PMMarvin
10/09/2025, 7:38 PMprefect-3:latest image
- Flow events appear in Cloud but flow logs do not
- CloudWatch shows the expected log lines (both print and get_run_logger())
This almost always points to the Prefect “logs to API” handler being disabled or overridden in the flow container, or to logs being dropped before they flush.
Things to check first (quick wins)
- Make sure these env vars are set in the JOB CONTAINER (not just the worker):
- PREFECT_LOGGING_TO_API_ENABLED=true
- PREFECT_LOGGING_LOG_PRINTS=true (since you want print lines too)
- PREFECT_LOGGING_LEVEL=INFO (or DEBUG)
- Verify the Cloud UI log filters (level, time range, “Show logs” toggles) aren’t hiding messages.
- If your flow completes very quickly, logs may not flush before the process exits. For a test run, add a short sleep at the end or reduce the batch interval (see below).
Where to set these for ECS runs
- Preferred: in your ECS work pool’s base job template (so they apply to every task the worker launches)
- Cloud UI: Work Pools -> your ECS pool -> Edit base job template -> Environment variables
- CLI (to inspect current values):
prefect work-pool inspect "<your-ecs-pool>"
Look for job_variables and the env section.
- Alternatively: set env vars in your deployment’s job variables/infrastructure overrides so they propagate to the task.
Common causes we see
- PREFECT_LOGGING_TO_API_ENABLED accidentally set to false in the work pool’s base job template or deployment overrides. This will give you events (they use the events service) but no logs in Cloud (they use the logs endpoint).
- A custom logging configuration overriding the default Prefect handlers
- If you set PREFECT_LOGGING_CONFIG_PATH (or configured a custom logging dict), it can remove the Prefect API log handler. Try unsetting it or ensure the Prefect API log handler is included.
- Logs dropped before flush
- By default logs are batched and flushed on a short interval. If the container exits right away, some logs may not make it.
- For testing, set:
- PREFECT_LOGGING_TO_API_BATCH_INTERVAL=0.5 (flush more frequently)
- And/or add a time.sleep(2) at the end of the flow to confirm the flush occurs
Useful diagnostics
- Inspect the effective Prefect settings inside the flow container (you’ll see this in CloudWatch even if logs don’t reach Cloud):
from prefect.settings import get_current_settings
from prefect import flow, get_run_logger
import time, os
@flow
def log_probe():
s = get_current_settings()
logger = get_run_logger()
<http://logger.info|logger.info>(f"Log level: {s.logging.level}")
<http://logger.info|logger.info>(f"Log prints: {s.logging.log_prints}")
<http://logger.info|logger.info>(f"To API enabled: {s.logging.to_api.enabled}")
<http://logger.info|logger.info>(f"Batch interval: {s.logging.to_api.batch_interval}")
<http://logger.info|logger.info>(f"Batch size: {s.logging.to_api.batch_size}")
<http://logger.info|logger.info>(f"Max log size: {s.logging.to_api.max_log_size}")
print("Print that should appear in Cloud logs if log_prints=True")
time.sleep(2) # give the API handler time to flush
if __name__ == "__main__":
log_probe()
- Turn on verbose internal logging to see if logs are being sent/dropped:
- PREFECT_LOGGING_LEVEL=DEBUG
- PREFECT_LOGGING_EXTRA_LOGGERS=prefect.client,prefect.logging
- Look in CloudWatch for warnings like “dropping log” or errors sending logs to the API.
Known issues
- We’re not tracking a 3.4.22-specific bug that causes flow logs to disappear from Cloud while events succeed. When this happens in ECS, it’s nearly always configuration (API logging disabled or overridden) or early container exit before the batch flush.
If you want, share:
- The output of prefect work-pool inspect "<your-ecs-pool>" (specifically the env in job_variables)
- Whether you set a custom PREFECT_LOGGING_CONFIG_PATHMarvin
10/09/2025, 7:38 PMPREFECT_LOGGING_* and PREFECT_LOGGING_TO_API_*)
- ECS worker guideJason Wiener
10/09/2025, 7:51 PM