Hey, I have have two questions about logging in Pr...
# ask-community
f
Hey, I have have two questions about logging in Prefect Orion (already read through the docs 1. Is there also an option to have all logs coming out of Docker, i.e. the Python process in the UI? I know, there’s
PREFECT_LOGGING_EXTRA_LOGGERS
and it works, but for example it does not cover logs, like coming from the Tensorflow CPP logs, as e.g.:
Copy code
2022-08-08 13:59:27.431966: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-08-08 13:59:27.432037: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Plus, it’s kind of tedious collecting all the libs I want to include. 2. When starting a flow run, many valuable logs and especially errors are only visible on the work queue. So if e.g. I misconfigured a block and the deployment run fails to start, an end user cannot see the error message, since the queue logs are not visible. E.g. one would need to check the work queue Kubernetes pod and manually filter out the error logs for yourself. A user might not have access and with many flow runs, this is also hard. Am I missing something, or how should one work with that properly?
k
I don’t know about this, but my teammates might. They’ll get back to you when they are available
f
Thank you so much 🙂
@Khuyen Tran Were you able to check with someone? 🙂
a
yes, there is! pass it as comma-delimited string
Copy code
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS='tensorflow,anotherone'
f
@Anna Geller That was not my question, I know this functionality 😄
1. So I figured out that the TF CPP logs are sent to stderr. Can one also forward stdout and stderr? 2. This one still remains completely open
a
We only capture logs, not stderr or stdout
f
So yes, and what about the logs of the work queue? Is there a way to make them accessible for a user?
a
The problem with this is that your agent is running on your infra, Prefect has no direct access to it to capture those agent-level logs/metrics directly from the machine
but nothing stops you to implement it yourself - if you run things on AWS, every compute service integrates with CloudWatch to capture logs
f
Yeah, capturing the logs would be possible, but then a user manually need to manually search for his deployment.
Hmmm, can I report these things as feature requests on Github? I think for production usage, these are very critical features
a
you may reach out to sales@prefect.io and we could help set this up but there is no default feature allowing to capture infra logs as of today
f
Thanks! 🙂 I can set up your suggestion without problems, but I don’t think it would solve the problem in a good way