https://prefect.io logo
Join Slack
Powered by
How do I get the k8s logs from a `RunNamespacedJob...
# prefect-server
d
How do I get the k8s logs from a 
RunNamespacedJob
task?
I see there is a 
ReadNamespacedPodLogs
and i found the PR that added it but I dont understand how to use it
n
Hi @Dana Merrick - you should be able to pass 
log_level
to your task to get logs from the job. (by default this is set to 
None
, which disables logging from the job)
d
I tried 
"debug"
for the log level but still get no output
does it have to be DEBUG or something case sensitive like that?
n
Hm no I think 
log_level='debug'
should work, can you send your task declaration?
d
yeah i can pick snippits out to show you...
Copy code
flow = s3_flow(sys.argv[0])
flow.add_task(run_command("ls -al"))

if __name__ == "__main__":
    flow.run()
Copy code
def run_command(command, image=None):
    # create a length-6 random string
    name = "".join(random.choices(string.ascii_uppercase + string.digits, k=6))
    image = image or DEFAULT_IMAGE

    body = {
        "apiVersion": "batch/v1",
        "kind": "Job",
        "metadata": {"name": name},
        "spec": {
            "template": {"spec": {"containers": [], "restartPolicy": "Never"}},
            "backoffLimit": 4,
        },
    }

    container = {"name": "executor", "image": image, "command": command.split(" ")}

    body["spec"]["template"]["spec"]["containers"].insert(0, container)

    return RunNamespacedJob(
        body=body,
        delete_job_after_completion=True,
        kubernetes_api_key_secret=False,
        log_level="debug",
        log_stdout=True,
        namespace="default",
    )
based heavily on Mariia's suggestion
n
Got it - it looks like you're returning the constructor from a task, is that true? If so, you'll need to call 
.run()
on it
d
not 100% sure what you mean there... if you're asking about 
s3_flow()
, it's:
Copy code
def s3_flow(flow_name, project_name=None, image=None):
    docker_image = image or DEFAULT_IMAGE
    project_name = project_name or DEFAULT_PROJECT

    flow = Flow(
        flow_name,
        run_config=KubernetesRun(labels=DEFAULT_LABELS, image=docker_image),
        storage=S3(bucket=DEFAULT_BUCKET),
    )
    return flow
the idea being most of our staff will need to plug the same stuff into Flow so we created a wrapper around it
the stuff in 
run_command()
does get run, it just gets swallowed
n
If 
run_command
is a task, its run method is being called by the flow runner; however since it's also returning a task (your, the runner doesn't know to call the 
run
method; try this:
Copy code
return RunNamespacedJob(
        body=body,
        delete_job_after_completion=True,
        kubernetes_api_key_secret=False,
        log_level="debug",
        log_stdout=True,
        namespace="default",
    ).run()
d
ah, cool, will do
still having issues with logs
Copy code
17:50:07
INFO
CloudTaskRunner
Task 'run_command': Starting task run...
17:50:07
INFO
RunNamespacedJob
Job 4au0dr has been created.
17:52:18
INFO
RunNamespacedJob
Started following logs for 4au0dr-ssc84
17:52:18
INFO
RunNamespacedJob
Job 4au0dr has been completed.
17:52:18
INFO
CloudTaskRunner
Task 'run_command': Finished task run for task with final state: 'Success'
this is the output for the logs view on the prefect UI
Copy code
total 140
drwxr-xr-x  1 root root   44 Mar 26 21:48 .
drwxr-xr-x  1 root root   17 Mar 26 21:52 ..
-rw-r--r--  1 root root  216 Mar 26 21:48 .dockerignore
drwxr-xr-x  2 root root    6 Mar 26 16:07 .empty
-rw-r--r--  1 root root   65 Mar 26 21:48 .gitattributes
-rw-r--r--  1 root root  257 Mar 26 21:48 .gitignore
...
this is the output for the pod on k8s
i'm basically just trying to get the k8s job output from my task to appear in the flow somewhere... anywhere...
n
Have you configured your default logging level for tasks? https://docs.prefect.io/core/concepts/logging.html#logging-configuration
If you haven't configured that, try passing 
log_level="info"
to your task constructor instead of 
debug
If your norma log level is set at
d
is the 
@task()
the task constructor?
or 
RunNamespacedJob()
n
Both are task constructors, in this case you'll want to pass that to 
RunNamespacedJob()
, as you were before but with a different log level
d
ah okay sure i can try that
no luck unfortunately
j
Hey, I authored that PR 🤓. I later found out that if you want to read the pod logs, the service account must have the ight role (RBAC) 
pods/log
If you use prrefect-server this recently was included in the helm chart, if not the role should look something like
Copy code
rules:
- apiGroups:
  - batch
  - extensions
  resources:
  - jobs
  - jobs/status
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - events
  - pods
  - pods/log
  - services
  verbs:
  - '*'
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - '*'
🙌 1
d
thank you! ill try this tomorrow
update: the RBAC permissions fixed my issue
🙌 1
ty!
j
Great! 🙂
2 Views
Open in Slack
PreviousNext
Powered by