https://prefect.io logo
Title
b

Ben Davison

06/09/2020, 11:44 AM
While trying to set log format in kubernetes for the scheduler, I've tried setting it like this:
- name: PREFECT__LOGGING__FORMAT
              value: '{"level": "%(levelname)s", "message": "%(message)s"}'
And I can see the pod has the environment variable set.
kubectl exec -it prefect-scheduler-6b96994c6-qtqh5 --namespace=data -- /bin/sh -c 'echo "password: $PREFECT__LOGGING__FORMAT"'                                                                                  <aws:default>
password: {"level": "%(levelname)s", "message": "%(message)s"}
But the logs are still in the default format:
kubectl --namespace data logs prefect-scheduler-6b96994c6-qtqh5 -f                                                                                                                                                                   <aws:default>
[2020-06-09 11:36:54,302] INFO - prefect-server.Scheduler | Scheduler will start after an initial delay of 275 seconds...
Does anyone have any idea? Or even better, I'm trying to get logs in datadog to be parsed correctly.
Just realised, is
PREFECT__LOGGING__FORMAT
just for tasks outputs?
ah, it's
PREFECT_SERVER__LOGGING__FORMAT
Interesting, even with:
- name: PREFECT_SERVER__LOGGING__FORMAT
              value: "{'level': '%(levelname)s', 'message': '%(message)s'}"
The output is blank:
kubectl --namespace data logs prefect-scheduler-649d8c4cb5-mz2pl -f                                                                                                                                                                  <aws:default>
^C
edit: deffo escaping issue
Strange:
- name: PREFECT_SERVER__LOGGING__FORMAT
              value: '{""level"": ""%(levelname)s"", ""message"": ""%(message)s""}'
Will show the logs (with "") I've tried escaping like so:
- name: PREFECT_SERVER__LOGGING__FORMAT
              value: '{\""level"\": \""%(levelname)s"\", \""message"\": \""%(message)s"\"}'
Same, and if I only do single " no logs show up atall.
yup will try, but I think it will have the same escaping issue
z

Zachary Hughes

06/09/2020, 1:06 PM
Sorry about that-- typed that before coffee and looks like you've already given it a shot. Taking a look into this as well!
One suggestion, and apologies if you've already tried it-- can you try specifying the logging format like this?
- name: PREFECT_SERVER__LOGGING__FORMAT
  value: "%(level) - %(name) | %(message)"
It looks like there might be a double-quoting issue in the snippets you posted.
b

Ben Davison

06/09/2020, 1:29 PM
yup that works, but I'm trying to get it to output json so datadog can parse the logs easily
ah didn't see the (remove s part)
ok, I'm convinced it's the
%
Doesn't work
value: '{"level": "%(levelname)s"}'
Works (but has the double "" issue)
value: '{"level": ""%(levelname)s"}'
I've tried escaping both the quotes with
\
and the
%
also. It always seems to double up whatever I put there.
z

Zachary Hughes

06/09/2020, 2:21 PM
I'm seeing some weird behavior here, but I'm not sure that it's related to the %. If I provide the value below, which was a typo, I get what I think you're looking for with one trailing quote on the message value. If I remove the "typo," I don't see any logs.
'{\"level\": \"%(levelname)s\", \"message\": \"%(message)s\""}'
I'm going to poke around for just a bit more, but will open an issue if I don't find a solution in the near future. It seems to be related to parsing for the last key-- will keep you posted.
b

Ben Davison

06/09/2020, 2:25 PM
thanks Zachary
z

Zachary Hughes

06/09/2020, 2:39 PM
@Marvin open "Issues specifying JSON log formatting"
z

Zachary Hughes

06/09/2020, 2:40 PM
Issue opened! I think there is something hinky going on, so this will let us triage and track a fix. Thank you for letting us know about this, @Ben Davison