<@ULVA73B9P> I seem to miss the log on the task on...
# marvin-ai
i
@Marvin I seem to miss the log on the task on prefect version 3.4.25. What are the possible reasons
m
thought for 139.8 seconds
Possible reasons in Prefect 3.4.25 that task logs don’t show up: Most common causes 1. Logging to the API is disabled - Check
PREFECT_LOGGING_TO_API_ENABLED
- If false, logs won’t appear in Prefect UI/Cloud 2. Log level is too high - Prefect defaults to
INFO
- If your task logs at
DEBUG
, those messages are filtered out unless you lower the level - Relevant settings: -
PREFECT_LOGGING_LEVEL
-
PREFECT_LOGGING_LOGGERS_PREFECT_TASK_RUNS_LEVEL
3. You’re not logging from an active flow/task run context -
get_run_logger()
needs a flow/task run context - If called outside a task/flow, logs may be dropped or warn depending on config 4. You’re using
print()
instead of the logger
-
print()
is only captured if
log_prints=True
- You can enable it on the flow/task:
Copy code
@flow(log_prints=True)
   @task(log_prints=True)
- Or globally with
PREFECT_LOGGING_LOG_PRINTS=True
5. Custom logging config removed Prefect’s API handler - If using a custom
logging.yml
, it may not attach
APILogHandler
to
prefect.task_runs
- Then logs may appear locally but not in the Prefect UI 6. Subprocess / process-boundary issues - If tasks run in separate processes, logs need context propagation and forwarding - This can affect: -
ProcessPoolTaskRunner
- manually spawned processes - custom multiprocessing code 7. Third-party logger vs Prefect run logger - If you use a normal Python logger instead of
get_run_logger()
, it may not go to task logs unless configured - Third-party library logs may also need
PREFECT_LOGGING_EXTRA_LOGGERS
8. Logs are sent, but not visible yet - Prefect batches API logs before sending - There can be a short delay before they appear in UI 9. Logs were truncated or dropped - Large log lines may be truncated - In very noisy runs, batching/queue issues can affect delivery 10. Worker/runtime environment issues - The worker environment may override logging settings - Remote execution environments may not match your local config Good checks to run
Copy code
prefect config view | grep LOGGING
Check specifically for: -
PREFECT_LOGGING_TO_API_ENABLED
-
PREFECT_LOGGING_LEVEL
-
PREFECT_LOGGING_LOG_PRINTS
Also compare: - logs in the UI - logs from CLI:
Copy code
prefect flow-run logs <FLOW_RUN_ID>
If you want, I can help you narrow it down quickly if you share: - whether you use
get_run_logger()
or
print()
- whether this happens for all tasks or only some - whether you run locally, in Docker, Kubernetes, or a worker - whether you use a special task runner like process pool Relevant docs: - Logging customization - Add logging to flows and tasks
i
@Marvin without any changes in configuration I sometimes receive the logs, but also task may finish in state Completed and not log. Why is that possible
m
thought for 133.1 seconds
Yes — that is possible, even without config changes. A task can end in Completed but still show no task logs because task state and task logs are separate signals. Prefect can record the final state successfully even when no log records are emitted, delayed, or lost before being flushed. Most likely explanations: 1. The task completed but never actually logged anything - A task only has task logs if your code emitted logs - If the task has no
<http://logger.info|logger.info>(...)
and no
print()
with
log_prints=True
, then Completed with no logs is expected 2. The task finished too quickly and logs were still buffered - Prefect batches logs before sending them to the API - So a fast task can: - send its Completed state immediately - but its logs are still waiting in the log batch - In some runtimes, those logs may appear later - In others, if the process exits quickly enough, they may never make it 3. The task ran in another process This is a big one if you use: -
ProcessPoolTaskRunner
- multiprocessing - subprocess-based execution In those cases, the task state can come back correctly, but logs from the subprocess may not be flushed/forwarded in time before that subprocess exits. 4. The task result came from cache / reused execution path - If Prefect uses a cached result, the task may not really execute user code again - That means it can appear done without producing fresh logs 5.
print()
is not the same as logging
- If you rely on
print()
, those messages only become Prefect logs if
log_prints=True
- Otherwise you may think “the task should have logged”, but Prefect never treated that output as task logs 6. The state update succeeded but log delivery was best-effort - State transitions are core orchestration data - Logs are sent through a separate API logging path - So intermittent network/API timing issues can affect logs without affecting the final task state 7. The task run existed, but logging context was missing at the moment of logging - If code logs outside the active task run context, Prefect may drop those records from task logs - The task can still complete normally So the short version is: “Completed” means the task execution/state was recorded successfully. It does not guarantee that: - logs were emitted - logs were captured - logs were flushed - logs were delivered before process shutdown What I’d check first 1. Are you using
get_run_logger()
or
print()
? 2. Is the task very fast? 3. Are you using process-based parallelism? 4. Does it happen more for small/short tasks than long ones? 5. If you wait a few seconds and refresh, do the logs appear later? A minimal reliable pattern is:
Copy code
from prefect import task, flow, get_run_logger

@task
def my_task():
    logger = get_run_logger()
    <http://logger.info|logger.info>("task started")
    return 42

@flow
def my_flow():
    my_task()

if __name__ == "__main__":
    my_flow()
If you want, I can help you pinpoint the exact cause if you share: - your task code - whether you use
ThreadPoolTaskRunner
or
ProcessPoolTaskRunner
- whether the missing logs are from
<http://logger.info|logger.info>(...)
or
print()
- where the flow is running (local worker, Docker, Kubernetes, etc.)