< Marvin> I am looking to print pod logs in prefect UI when Prefect Community #ask-marvin

<@ULVA73B9P> I am looking to print pod logs in pre...

Pooyan

08/01/2024, 7:11 PM

@Marvin I am looking to print pod logs in prefect UI when running a kubernetes job. I am trying run_namespaced_job(job, print_func=print) but it is not printing anything interactively in the UI. I am using prefect_kubernetes==0.3.11

Marvin

08/01/2024, 7:13 PM

It seems like you're encountering an issue with logging pod output directly to the Prefect UI when using the

run_namespaced_job

function in conjunction with a Kubernetes job. ### Key Points from Documentation and Issues: 1. Enabling Print Logging Globally: To ensure print statements are captured, you might need to configure logging globally for all flows and tasks. This can be done using:

Copy code

prefect config set PREFECT_LOGGING_LOG_PRINTS=True

Reference: Logging Print Statements 2. Using
read_namespaced_pod_logs
: The

read_namespaced_pod_logs

function in

prefect_kubernetes

can be used to read and print pod logs. Here is an example:

Copy code

python
   from prefect import flow, get_run_logger
   from prefect_kubernetes.credentials import KubernetesCredentials
   from prefect_kubernetes.pods import read_namespaced_pod_logs

   @flow
   def kubernetes_orchestrator():
       logger = get_run_logger()
       pod_logs = read_namespaced_pod_logs(
           kubernetes_credentials=KubernetesCredentials.load("k8s-creds"),
           pod_name="test-pod",
           container="test-container",
           print_func=<http://logger.info|logger.info>
       )
       return pod_logs

Reference: Prefect Kubernetes Pods 3. Known Issues with Logging: There are recognized issues where Kubernetes job logs may not display correctly in the Prefect UI. A specific issue addressing this problem is logged on GitHub: - Issue: Kubernetes jobs' log couldn't print in prefect UI - This issue indicates that there can be timing problems when logs are not immediately available, causing them not to be captured. ### Recommendations: 1. Configure Logging Globally: Ensure that logging of print statements is enabled globally across all Prefect tasks and flows by setting the relevant configuration. 2. Reading Pod Logs: Use the

read_namespaced_pod_logs

function as shown in the example to manually capture and log pod output. 3. Monitor GitHub Issues: Keep an eye on the linked GitHub issues for updates or potential fixes related to logging in Kubernetes jobs. Hopefully, these steps should help you resolve or work around the issue you are facing.

Pooyan

08/01/2024, 10:11 PM

@Marvin thanks. read_namespaced_pod_log is a good option but the problem is pod disappears by the time run_namespaced_job completes so when I am looking to get the pod names using CoreV1Api.list_namespaced_pod(namespace, label_selector=f"job-name={job_name}"), job has already completed and logs are gone

Marvin

08/01/2024, 10:11 PM

It seems like you're encountering a timing issue where the pods disappear by the time the job completes, making it difficult to fetch the logs. Here's a potential solution to tackle this problem: ### Suggested Approach 1. Using
fetch_result
method: The

fetch_result

method in

<http://prefect_kubernetes.jobs|prefect_kubernetes.jobs>

can be used to retrieve the logs of completed jobs. This method captures logs from each pod in the job and can be fetched even after the job has completed. Here is an example of how to use the

fetch_result

method:

Copy code

python
from prefect import flow, get_run_logger
from prefect_kubernetes import KubernetesJob, KubernetesCredentials
from <http://prefect_kubernetes.jobs|prefect_kubernetes.jobs> import run_namespaced_job

@flow
def kube_job_flow():
    job = KubernetesJob(
        namespace="default",
        job_manifest_path="path/to/job_manifest.yaml",
        delete_after_completion=True  # Ensures cleanup after fetching results
    )

    # Run the Kubernetes job
    job_run = run_namespaced_job(
        kubernetes_credentials=KubernetesCredentials.load("k8s-creds"),
        job=job
    )

    logger = get_run_logger()
    
    # Wait for the job to complete and fetch the results (including logs)
    result = job_run.fetch_result()
    for pod_name, logs in result.items():
        <http://logger.info|logger.info>(f"Logs from pod {pod_name}:\n{logs}")

kube_job_flow()

### Important Considerations - Ensure

delete_after_completion

is set to

True

for the job to automatically clean up after fetching the logs. - The

fetch_result

method can be called after the namespace job is completed to fetch logs from the terminated pods. ### References 1. Prefect Kubernetes Jobs 2. Prefect Logs Guide Hopefully, this approach helps you to reliably fetch pod logs in the Prefect UI even after the jobs complete.

22 Views

Open in Slack

Previous Next