https://prefect.io logo
Channels
Powered by
d

Dana Merrick

03/26/2021, 9:18 PM
How do I get the k8s logs from a 
RunNamespacedJob
task?
I see there is a 
ReadNamespacedPodLogs
and i found the PR that added it but I dont understand how to use it
n

nicholas

03/26/2021, 9:20 PM
Hi @Dana Merrick - you should be able to pass 
log_level
to your task to get logs from the job. (by default this is set to 
None
, which disables logging from the job)
d

Dana Merrick

03/26/2021, 9:20 PM
I tried 
"debug"
for the log level but still get no output
does it have to be DEBUG or something case sensitive like that?
n

nicholas

03/26/2021, 9:26 PM
Hm no I think 
log_level='debug'
should work, can you send your task declaration?
d

Dana Merrick

03/26/2021, 9:26 PM
yeah i can pick snippits out to show you...
Copy code
flow = s3_flow(sys.argv[0])
flow.add_task(run_command("ls -al"))

if __name__ == "__main__":
    flow.run()
Copy code
def run_command(command, image=None):
    # create a length-6 random string
    name = "".join(random.choices(string.ascii_uppercase + string.digits, k=6))
    image = image or DEFAULT_IMAGE

    body = {
        "apiVersion": "batch/v1",
        "kind": "Job",
        "metadata": {"name": name},
        "spec": {
            "template": {"spec": {"containers": [], "restartPolicy": "Never"}},
            "backoffLimit": 4,
        },
    }

    container = {"name": "executor", "image": image, "command": command.split(" ")}

    body["spec"]["template"]["spec"]["containers"].insert(0, container)

    return RunNamespacedJob(
        body=body,
        delete_job_after_completion=True,
        kubernetes_api_key_secret=False,
        log_level="debug",
        log_stdout=True,
        namespace="default",
    )
based heavily on Mariia's suggestion
n

nicholas

03/26/2021, 9:29 PM
Got it - it looks like you're returning the constructor from a task, is that true? If so, you'll need to call 
.run()
on it
d

Dana Merrick

03/26/2021, 9:30 PM
not 100% sure what you mean there... if you're asking about 
s3_flow()
, it's:
Copy code
def s3_flow(flow_name, project_name=None, image=None):
    docker_image = image or DEFAULT_IMAGE
    project_name = project_name or DEFAULT_PROJECT

    flow = Flow(
        flow_name,
        run_config=KubernetesRun(labels=DEFAULT_LABELS, image=docker_image),
        storage=S3(bucket=DEFAULT_BUCKET),
    )
    return flow
the idea being most of our staff will need to plug the same stuff into Flow so we created a wrapper around it
the stuff in 
run_command()
does get run, it just gets swallowed
n

nicholas

03/26/2021, 9:35 PM
If 
run_command
is a task, its run method is being called by the flow runner; however since it's also returning a task (your, the runner doesn't know to call the 
run
method; try this:
Copy code
return RunNamespacedJob(
        body=body,
        delete_job_after_completion=True,
        kubernetes_api_key_secret=False,
        log_level="debug",
        log_stdout=True,
        namespace="default",
    ).run()
d

Dana Merrick

03/26/2021, 9:35 PM
ah, cool, will do
still having issues with logs
Copy code
17:50:07
INFO
CloudTaskRunner
Task 'run_command': Starting task run...
17:50:07
INFO
RunNamespacedJob
Job 4au0dr has been created.
17:52:18
INFO
RunNamespacedJob
Started following logs for 4au0dr-ssc84
17:52:18
INFO
RunNamespacedJob
Job 4au0dr has been completed.
17:52:18
INFO
CloudTaskRunner
Task 'run_command': Finished task run for task with final state: 'Success'
this is the output for the logs view on the prefect UI
Copy code
total 140
drwxr-xr-x  1 root root   44 Mar 26 21:48 .
drwxr-xr-x  1 root root   17 Mar 26 21:52 ..
-rw-r--r--  1 root root  216 Mar 26 21:48 .dockerignore
drwxr-xr-x  2 root root    6 Mar 26 16:07 .empty
-rw-r--r--  1 root root   65 Mar 26 21:48 .gitattributes
-rw-r--r--  1 root root  257 Mar 26 21:48 .gitignore
...
this is the output for the pod on k8s
i'm basically just trying to get the k8s job output from my task to appear in the flow somewhere... anywhere...
n

nicholas

03/26/2021, 10:36 PM
Have you configured your default logging level for tasks? https://docs.prefect.io/core/concepts/logging.html#logging-configuration
If you haven't configured that, try passing 
log_level="info"
to your task constructor instead of 
debug
If your norma log level is set at
d

Dana Merrick

03/26/2021, 10:45 PM
is the 
@task()
the task constructor?
or 
RunNamespacedJob()
n

nicholas

03/26/2021, 10:49 PM
Both are task constructors, in this case you'll want to pass that to 
RunNamespacedJob()
, as you were before but with a different log level
d

Dana Merrick

03/26/2021, 10:49 PM
ah okay sure i can try that
no luck unfortunately
j

Joël Luijmes

03/27/2021, 6:43 PM
Hey, I authored that PR 🤓. I later found out that if you want to read the pod logs, the service account must have the ight role (RBAC) 
pods/log
If you use prrefect-server this recently was included in the helm chart, if not the role should look something like
Copy code
rules:
- apiGroups:
  - batch
  - extensions
  resources:
  - jobs
  - jobs/status
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - events
  - pods
  - pods/log
  - services
  verbs:
  - '*'
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - '*'
🙌 1
d

Dana Merrick

03/28/2021, 4:04 PM
thank you! ill try this tomorrow
update: the RBAC permissions fixed my issue
🙌 1
ty!
j

Joël Luijmes

04/06/2021, 9:49 AM
Great! 🙂
Powered by