<@ULVA73B9P> I cant see the logs from a finished k...
# ask-marvin
h
@Marvin I cant see the logs from a finished kubernetes worker
m
thought for 166.9 seconds
I can help troubleshoot! A couple quick clarifiers first: - Are you missing logs for the flow run in the Prefect UI, or the Kubernetes pod logs for the worker/job in your cluster? - Are you on Prefect Cloud or Prefect Server? And which work pool type are you using (Kubernetes, Process, etc.)? In Prefect 3.x, there are two kinds of logs to consider: 1) Flow run logs — emitted by your flow code and sent to the Prefect API (these should remain visible after the run finishes if they were successfully uploaded) 2) Kubernetes pod logs — stdout/stderr from the worker or job pod, visible via kubectl or your cluster logging stack (these can disappear if the pod is cleaned up) Common reasons logs are missing and how to fix them: - The pod was deleted after completion - If you need kubectl logs after completion, keep the Kubernetes Job around by setting a TTL: - In a Kubernetes work pool’s base job template, add a TTL on the Job:
Copy code
apiVersion: batch/v1
      kind: Job
      spec:
        ttlSecondsAfterFinished: 86400   # keep for 1 day
        template:
          spec:
            containers:
              - name: flow
                ...
- Then you can fetch logs with:
Copy code
kubectl logs job/<job-name> -n <namespace>
    # or by pod:
    kubectl logs <pod-name> -n <namespace>
- The pod could not send logs to Prefect - Verify the job/worker has network egress to the Prefect API and correct env vars: -
PREFECT_API_URL
(and
PREFECT_API_KEY
for Cloud) - If the run finished but you see no flow logs in the UI, this is often a connectivity issue. - Logging level is too high - Ensure logs aren’t suppressed. In 3.x you can lower the root log level via: -
PREFECT_LOGGING_ROOT_LEVEL=INFO
(or DEBUG) - Also make sure your code is actually logging:
Copy code
from prefect import flow, get_run_logger

    @flow
    def demo():
        logger = get_run_logger()
        <http://logger.info|logger.info>("Hello from the flow!")
- You’re looking for worker logs vs. flow run logs - Flow run logs are on the Flow Run page in the UI. - Worker logs (the long-running process started with
prefect worker start
) are best viewed via your cluster logs:
Copy code
kubectl logs deploy/<worker-deployment> -n <namespace>
In the Prefect UI, Work Pools > Workers can show recent worker logs, but long-term retention relies on your cluster logging. Quick ways to locate the right pod: - By job name (if you know it):
kubectl logs job/<job-name> -n <ns>
- By labels (replace with your flow run id):
kubectl get pods -n <ns> -l <http://prefect.io/flow-run-id=<FLOW_RUN_ID|prefect.io/flow-run-id=<FLOW_RUN_ID>>
Relevant docs: - Prefect logs API and retrieval: Read logs via API - Logging customization in 3.x: Logging customization - Kubernetes deployment/job guidance: Kubernetes how-to If you can share: - A link or ID for an example flow run with missing logs - Your work pool type and base job template (if using a Kubernetes work pool) - Whether the pod still exists (output of
kubectl get pods -n <ns>
) …I can pinpoint the exact cause and suggest a precise fix.