https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> what are the best practices for switc...
# ask-marvin
a
@Marvin what are the best practices for switching between python logging package logger for developing libraries and non-prefect code, and prefect logger when those libraries are used in prefect flows
m
Here are some best practices for integrating Python's logging package with Prefect's logging system: 1. Configuration Management: - Prefect allows logging configuration to be customized through environment variables or Prefect Profile settings. - You can override any logging setting using the syntax 
PREFECT_LOGGING_[PATH]_[TO]_[KEY]
. - Prefect provides a default 
logging.yml
file, and you can create your own version with custom settings. 2. Log Level Management: - Adjust log levels for Prefect using the environment variable 
PREFECT_LOGGING_LEVEL
. - Configure the root Python logger, which Prefect uses for logging, to a desired level, e.g., 
PREFECT_LOGGING_ROOT_LEVEL
. - Set specific handler log levels, e.g., 
PREFECT_LOGGING_HANDLERS_API_LEVEL
, to control what gets reported to Prefect API. 3. Combining Loggers: - By default, the root logger receives logs from all loggers unless propagation is explicitly disabled. - Ensure that loggers from other libraries have their log levels adjusted to integrate seamlessly with Prefect logs. 4. Deployment Considerations: - For deploying Prefect flows, consider utilizing the new deployment schemas and configurations supported by Prefect's latest updates. For further details, refer to the Prefect logging guide here. You might also want to keep an eye on updates and related discussions such as this one regarding deployment and logging best practices. If you have any specific scenarios or more questions, feel free to ask!
a
@Marvin as an example, I have a module 
test_lib
that defines a 
logger = getLogger(__name__)
as a global variable that is then used by a 
test
function; in another file I have a flow that uses the 
test
function, but I want to use the prefect logger instead of the python one when I run this function inside a flow. how can I do that?
m
To switch between the Python logging package's logger and the Prefect logger when using a library inside Prefect flows, you can adjust your logging setup as follows: 1. Library Code: - Wrap the logging setup in a function that allows injecting the logger, making it flexible to switch loggers.
Copy code
python
# test_lib.py
import logging

def get_logger(name=__name__):
    return logging.getLogger(name)

logger = get_logger()

def test():
    <http://logger.info|logger.info>("This is a test log message from test_lib.")
2. Flow Code: - Use 
prefect.get_run_logger
to get the Prefect logger within your flow or task. - Override the logger in the library function accordingly.
Copy code
python
# flow_file.py
from prefect import flow, get_run_logger
import test_lib

@flow(name="test-flow")
def my_flow():
    prefect_logger = get_run_logger()
    
    # Inject the Prefect logger into the library
    test_lib.logger = prefect_logger
    
    # Call the library function
    test_lib.test()

if __name__ == "__main__":
    my_flow()
Here's how the flow and custom logging integration works: - In the library (
test_lib
), set up a function to get a logger, allowing dynamic reassignment. - In the flow, use 
get_run_logger()
to obtain the Prefect logger and reassign it in the library before using it. For more detailed logging configurations, refer to the Prefect logging guide here. Feel free to ask if you have further questions or need more examples!
a
@Marvin this works, but I have to go and overwrite logger in every library util that I use, is there a way to streamline that?
m
To streamline the process of switching loggers without manually overwriting the logger in every library utility, you can create a custom logging setup function. This setup function will replace the library's logger with the Prefect logger dynamically. Here's an example approach: 1. Library Code (Unchanged): - Set up your logger as usual.
Copy code
python
# test_lib.py
import logging

def get_logger(name=__name__):
    return logging.getLogger(name)

logger = get_logger()

def test():
    <http://logger.info|logger.info>("This is a test log message from test_lib.")
2. Flow Code: - Create a helper function that sets up the logger for your library. - Use this helper function in your flow to switch to the Prefect logger seamlessly.
Copy code
python
# logger_setup.py
from prefect import get_run_logger

def setup_library_logger(module, logger_name="logger"):
    """
    Sets up a Prefect logger for the given module.
    
    Parameters:
    - module: The module where the logger needs to be set up.
    - logger_name: The name of the logger attribute in the module.
    """
    prefect_logger = get_run_logger()
    setattr(module, logger_name, prefect_logger)

# flow_file.py
from prefect import flow
import test_lib
from logger_setup import setup_library_logger

@flow(name="test-flow")
def my_flow():
    # Setup the Prefect logger in the library
    setup_library_logger(test_lib)
    
    # Call the library function
    test_lib.test()

if __name__ == "__main__":
    my_flow()
By using this setup: 1. 
logger_setup.py
: - A helper function 
setup_library_logger
dynamically changes the logger for any given module. 2. 
flow_file.py
: - Call 
setup_library_logger
for every library/module you want to use the Prefect logger with. This approach centralizes the logger setup process, making it more maintainable and reusable. You can now reuse 
setup_library_logger
for any other modules, keeping your code clean and modular. If you need more examples or have any further questions, feel free to ask!
n
be warned, I skimmed this, but there's a setting called PREFECT_LOGGING_EXTRA_LOGGERS
which takes space delimited names of packages that configure a python logger, e.g. 
langchain pandas
etc and any of those extra loggers' records would get scooped by the prefect logger, if thats what you're looking for
g
@artur
11 Views
Open in Slack
PreviousNext
Powered by