https://prefect.io logo
Join Slack
Powered by
While trying to set log format in kubernetes for t...
# prefect-community
b
While trying to set log format in kubernetes for the scheduler, I've tried setting it like this:
Copy code
- name: PREFECT__LOGGING__FORMAT
              value: '{"level": "%(levelname)s", "message": "%(message)s"}'
And I can see the pod has the environment variable set.
Copy code
kubectl exec -it prefect-scheduler-6b96994c6-qtqh5 --namespace=data -- /bin/sh -c 'echo "password: $PREFECT__LOGGING__FORMAT"'                                                                                  <aws:default>
password: {"level": "%(levelname)s", "message": "%(message)s"}
But the logs are still in the default format:
Copy code
kubectl --namespace data logs prefect-scheduler-6b96994c6-qtqh5 -f                                                                                                                                                                   <aws:default>
[2020-06-09 11:36:54,302] INFO - prefect-server.Scheduler | Scheduler will start after an initial delay of 275 seconds...
Does anyone have any idea? Or even better, I'm trying to get logs in datadog to be parsed correctly.
Just realised, is 
PREFECT__LOGGING__FORMAT
just for tasks outputs?
ah, it's 
PREFECT_SERVER__LOGGING__FORMAT
Interesting, even with:
Copy code
- name: PREFECT_SERVER__LOGGING__FORMAT
              value: "{'level': '%(levelname)s', 'message': '%(message)s'}"
The output is blank:
Copy code
kubectl --namespace data logs prefect-scheduler-649d8c4cb5-mz2pl -f                                                                                                                                                                  <aws:default>
^C
edit: deffo escaping issue
Strange:
Copy code
- name: PREFECT_SERVER__LOGGING__FORMAT
              value: '{""level"": ""%(levelname)s"", ""message"": ""%(message)s""}'
Will show the logs (with "") I've tried escaping like so:
Copy code
- name: PREFECT_SERVER__LOGGING__FORMAT
              value: '{\""level"\": \""%(levelname)s"\", \""message"\": \""%(message)s"\"}'
Same, and if I only do single " no logs show up atall.
yup will try, but I think it will have the same escaping issue
z
Sorry about that-- typed that before coffee and looks like you've already given it a shot. Taking a look into this as well!
One suggestion, and apologies if you've already tried it-- can you try specifying the logging format like this?
Copy code
- name: PREFECT_SERVER__LOGGING__FORMAT
  value: "%(level) - %(name) | %(message)"
It looks like there might be a double-quoting issue in the snippets you posted.
b
yup that works, but I'm trying to get it to output json so datadog can parse the logs easily
ah didn't see the (remove s part)
ok, I'm convinced it's the 
%
Doesn't work 
value: '{"level": "%(levelname)s"}'
Works (but has the double "" issue) 
value: '{"level": ""%(levelname)s"}'
I've tried escaping both the quotes with 
\
and the 
%
also. It always seems to double up whatever I put there.
z
I'm seeing some weird behavior here, but I'm not sure that it's related to the %. If I provide the value below, which was a typo, I get what I think you're looking for with one trailing quote on the message value. If I remove the "typo," I don't see any logs. 
'{\"level\": \"%(levelname)s\", \"message\": \"%(message)s\""}'
I'm going to poke around for just a bit more, but will open an issue if I don't find a solution in the near future. It seems to be related to parsing for the last key-- will keep you posted.
b
thanks Zachary
z
@Marvin open "Issues specifying JSON log formatting"
m
z
Issue opened! I think there is something hinky going on, so this will let us triage and track a fix. Thank you for letting us know about this, @Ben Davison
Open in Slack
PreviousNext
Powered by