https://prefect.io logo
Title
f

Felix Sonntag

08/08/2022, 2:11 PM
Hey, I have have two questions about logging in Prefect Orion (already read through the docs 1. Is there also an option to have all logs coming out of Docker, i.e. the Python process in the UI? I know, there’s
PREFECT_LOGGING_EXTRA_LOGGERS
and it works, but for example it does not cover logs, like coming from the Tensorflow CPP logs, as e.g.:
2022-08-08 13:59:27.431966: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-08-08 13:59:27.432037: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Plus, it’s kind of tedious collecting all the libs I want to include. 2. When starting a flow run, many valuable logs and especially errors are only visible on the work queue. So if e.g. I misconfigured a block and the deployment run fails to start, an end user cannot see the error message, since the queue logs are not visible. E.g. one would need to check the work queue Kubernetes pod and manually filter out the error logs for yourself. A user might not have access and with many flow runs, this is also hard. Am I missing something, or how should one work with that properly?
k

Khuyen Tran

08/08/2022, 2:50 PM
I don’t know about this, but my teammates might. They’ll get back to you when they are available
f

Felix Sonntag

08/09/2022, 1:56 AM
Thank you so much 🙂
@Khuyen Tran Were you able to check with someone? 🙂
a

Anna Geller

08/10/2022, 4:22 PM
yes, there is! pass it as comma-delimited string
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS='tensorflow,anotherone'
f

Felix Sonntag

08/11/2022, 7:51 AM
@Anna Geller That was not my question, I know this functionality 😄
1. So I figured out that the TF CPP logs are sent to stderr. Can one also forward stdout and stderr? 2. This one still remains completely open
a

Anna Geller

08/11/2022, 10:08 AM
We only capture logs, not stderr or stdout
f

Felix Sonntag

08/11/2022, 10:26 AM
So yes, and what about the logs of the work queue? Is there a way to make them accessible for a user?
a

Anna Geller

08/11/2022, 10:40 AM
The problem with this is that your agent is running on your infra, Prefect has no direct access to it to capture those agent-level logs/metrics directly from the machine
but nothing stops you to implement it yourself - if you run things on AWS, every compute service integrates with CloudWatch to capture logs
f

Felix Sonntag

08/11/2022, 10:50 AM
Yeah, capturing the logs would be possible, but then a user manually need to manually search for his deployment.
Hmmm, can I report these things as feature requests on Github? I think for production usage, these are very critical features
a

Anna Geller

08/11/2022, 10:53 AM
you may reach out to sales@prefect.io and we could help set this up but there is no default feature allowing to capture infra logs as of today
f

Felix Sonntag

08/11/2022, 10:56 AM
Thanks! 🙂 I can set up your suggestion without problems, but I don’t think it would solve the problem in a good way