Thread
#prefect-community
    Tony Yun

    Tony Yun

    11 months ago
    Hi! I’m having some trouble in getting proper logs from 
    RunNamespacedJob
     task. I had set the 
    log_level='info'
     option, but when it’s failed, no any logs are being sent to Prefect. Instead, I only get this from UI.
    FAIL signal raised: FAIL('Job dbt-run-from-flow failed, check Kubernetes pod logs for more information.')
    After going to the k8s pod, I see exception logs:
    dbt ls --models tm_snowflake.* --profiles-dir=. --profile default --target dev
    Encountered an error while reading profiles:
      ERROR Runtime Error
      Compilation Error
        Could not render {{ env_var('DBT_PASSWORD') }}: Env var required but not provided: 'DBT_PASSWORD'
    Encountered an error:
    Runtime Error
      Could not run dbt
    make: *** [Makefile:6: ls] Error 2
    How could I pass any logs from k8s pods to Prefect UI?
    Kevin Kho

    Kevin Kho

    11 months ago
    Hey @Tony Yun, I think there is also a
    ReadNamespacedPodLogs
    task you have to use to pull the logs
    Tony Yun

    Tony Yun

    11 months ago
    oh I see. How could I miss this one.. thanks!
    Kevin Kho

    Kevin Kho

    11 months ago
    Actually no, I think it should do it for you here
    Maybe you can try with debug level?
    Tony Yun

    Tony Yun

    11 months ago
    hmm. debug gives me too much extra which I don’t like
    Kevin Kho

    Kevin Kho

    11 months ago
    I guess you might need to list the pods, find it, then read the logs?
    Tony Yun

    Tony Yun

    11 months ago
    ok let me try few options
    debug level doesn’t work either
    Kevin Kho

    Kevin Kho

    11 months ago
    Looks like there is a bug with that here
    There is also a workaround. Feel free to chime in also there
    Tony Yun

    Tony Yun

    11 months ago
    i think that’s exactly what’s happened
    ok then
    btw, there is also another bug to me that, the Prefect code hard coded k8s job names so I’m not allowed to use
    generateName
    to generate Random k8s job names:https://github.com/PrefectHQ/prefect/blob/8fcb9aa97d739fa25405c60cbb28b649c130bd3e/src/prefect/tasks/kubernetes/job.py#L706
    should I report this in Issues too?
    Kevin Kho

    Kevin Kho

    11 months ago
    Won’t the name already be generated by then?
    Ah I see what you mean
    Yeah feel free to open a separate issue for that?
    Tony Yun

    Tony Yun

    11 months ago
    ok
    it’s sad to me that I couldn’t make k8s pod logs available on Prefect… which the job will run blindly and requires manual intervention on CLI to view logs..
    Kevin Kho

    Kevin Kho

    11 months ago
    Hey I heard you will have an enterprise support channel. Those guys will be able to help better haha
    Tony Yun

    Tony Yun

    11 months ago
    hah, I’m not sure if it’s better than this channel. let me see.
    Kevin Kho

    Kevin Kho

    11 months ago
    Oh they will be because of this