Hey, I have have two questions about logging in Prefect Orion (already read through the docs 1. Is there also an option to have all logs coming out of Docker, i.e. the Python process in the UI? I know, there’s

PREFECT_LOGGING_EXTRA_LOGGERS

and it works, but for example it does not cover logs, like coming from the Tensorflow CPP logs, as e.g.:

2022-08-08 13:59:27.431966: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-08-08 13:59:27.432037: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

Plus, it’s kind of tedious collecting all the libs I want to include. 2. When starting a flow run, many valuable logs and especially errors are only visible on the work queue. So if e.g. I misconfigured a block and the deployment run fails to start, an end user cannot see the error message, since the queue logs are not visible. E.g. one would need to check the work queue Kubernetes pod and manually filter out the error logs for yourself. A user might not have access and with many flow runs, this is also hard. Am I missing something, or how should one work with that properly?

I don’t know about this, but my teammates might. They’ll get back to you when they are available

Thank you so much 🙂

yes, there is! pass it as comma-delimited string

prefect config set PREFECT_LOGGING_EXTRA_LOGGERS='tensorflow,anotherone'

@Anna Geller That was not my question, I know this functionality 😄

We only capture logs, not stderr or stdout

So yes, and what about the logs of the work queue? Is there a way to make them accessible for a user?

The problem with this is that your agent is running on your infra, Prefect has no direct access to it to capture those agent-level logs/metrics directly from the machine

Yeah, capturing the logs would be possible, but then a user manually need to manually search for his deployment.

you may reach out to sales@prefect.io and we could help set this up but there is no default feature allowing to capture infra logs as of today

Thanks! 🙂 I can set up your suggestion without problems, but I don’t think it would solve the problem in a good way