https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> How to add logs from an external pyth...
# ask-marvin
a
@Marvin How to add logs from an external python library (dbt) ?
@Marvin Please our prod is down
m
Here are some resources that might help you with logging from an external Python library, specifically dbt: 1. Events and logs | dbt Developer Hub: This page talks about structured logging and the eventing system in dbt-core. Note that the EVENT_HISTORY object has been deprecated and removed in dbt Core v1.4+. 2. Debug / log output from dbt python models - Stack Overflow: This Stack Overflow thread discusses how to view variables/output from Python models in dbt. It mentions a workaround involving replicating the Python model in a Jupyter notebook due to the lack of print() support. 3. Python models | dbt Developer Hub: This page provides an overview of dbt Python (dbt-py) models, which can help you solve use cases that can't be solved with SQL. You can perform analyses using tools available in the open-source Python ecosystem, including state-of-the-art packages for data science and statistics. Remember, life is full of small disappointments, like the fact that dbt's oversight doesn't include your Python model's print() statements. But don't worry, you'll get through this. You always do.
a
@Marvin Isn't there an extra_loggers functionality that can be used?
m
I'm sorry to hear that your production environment is down. As a chatbot, I don't have direct access to your systems, but I can guide you through some general steps to troubleshoot the issue: 1. Check the Logs: The first step in any troubleshooting is to check the logs. They can provide valuable information about what went wrong. 2. Monitor System Metrics: Check your system's CPU usage, memory usage, disk usage, and network traffic. These metrics can often point to the source of the problem. 3. Check Recent Changes: Have there been any recent deployments or changes to the production environment? If so, these could be the source of the problem. 4. Test Connectivity: If your production environment is hosted on a server, check to see if you can connect to it. If you can't, the problem could be network-related. 5. Contact Support: If you're still unable to resolve the issue, it might be time to contact your hosting provider's support team or a knowledgeable colleague. Remember, it's important to stay calm and systematic when dealing with production issues. Good luck!
Yes, you are correct. Prefect allows you to customize logging configuration by creating your own version of 
logging.yml
with custom settings. You can specify the path to this file with 
PREFECT_LOGGING_SETTINGS_PATH
. If the file does not exist at the specified location, Prefect ignores the setting and uses the default configuration. To access the Prefect logger, you can import it using 
from prefect import get_run_logger
. You can send messages to the logger in both flows and tasks. To log from a flow, retrieve a logger instance with 
get_run_logger()
, then call the standard Python logging methods. Here is an example:
Copy code
from prefect import flow, get_run_logger 

@flow(name="log-example-flow") 
def logger_flow(): 
    logger = get_run_logger() 
    <http://logger.info|logger.info>("INFO level log message.")
Prefect automatically uses the flow run logger based on the flow context. If you run the above code, Prefect captures the following as a log event. 
15:35:17.304 | INFO | Flow run 'mottled-marten' - INFO level log message.
The default flow run log formatter uses the flow run name for log messages. Starting in 2.7.11, if you use a logger that sends logs to the API outside of a flow or task run, a warning will be shown. You may need to change the log level used by loggers from other libraries to see their log records. You can override any logging configuration by setting an environment variable or Prefect configuration value. For more information, you can refer to the Prefect logging guide.
j
@Andreas Nord https://docs.prefect.io/2.14.2/guides/logs/#including-logs-from-other-libraries The 
PREFECT_LOGGING_EXTRA_LOGGERS
setting^
upvote 1
a
Copy code
PREFECT_LOGGING_EXTRA_LOGGERS=dbt
Does not work for me
n
>
Copy code
PREFECT_LOGGING_EXTRA_LOGGERS
that should be what you want, how are you trying to use it? here's an example that works for me the thing you pass should be the module
a
This is the code:
Copy code
import os

from prefect import flow
from dbt.cli.main import dbtRunner


@flow
def run_dbt():
    print(os.getenv("PREFECT_LOGGING_EXTRA_LOGGERS"))
    dbt_run = dbtRunner().invoke(["run"])


if __name__ == "__main__":
    run_dbt()
And the relevant part of the log (as you can see the last log message is not part of prefect logs):
Copy code
dbt
16:57:47.044 | DEBUG   | Flow run 'big-tench' - Executing flow 'run-dbt' for flow run 'big-tench'...
16:57:47.045 | DEBUG   | Flow run 'big-tench' - Beginning execution...
15:57:47  Running with dbt=1.6.6
4 Views
Open in Slack
PreviousNext
Powered by