Thread
#prefect-community
    jack

    jack

    3 months ago
    What would it take to have each task send logs both to the cloud and to a file on the local disk (A different file for each task)?
    Here is a snippet that attaches two log handlers to the root logger. Note
    configure_logging()
    is called outside of the task. If
    configure_logging()
    is moved inside the task, exceptions are raised.
    import logging
    import sys
    
    import prefect
    from prefect import Flow, task
    
    
    def configure_logging():
        #logger = prefect.utilities.logging.get_logger()
        logger = logging.getLogger()
    
        # If your logger is the root logger, you may want to set it to INFO or DEBUG
        logger.setLevel('INFO')
    
        handler = logging.FileHandler('mylog.log', encoding='utf-8')
        handler2 = logging.StreamHandler(sys.stdout)
    
        formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s',
                                      datefmt='%Y-%m-%d_%H:%M:%S')
    
        handler.setFormatter(formatter)
        handler2.setFormatter(formatter)
    
        logger.addHandler(handler)
        logger.addHandler(handler2)
    
    # configure logging outside of task
    configure_logging()
    
    @task(log_stdout=True)
    def log_something():
        <http://logging.info|logging.info>('Hello')
    
    with Flow('log-test') as flow:
        log_something()
    
    
    flow.run()
    Kevin Kho

    Kevin Kho

    3 months ago
    What error do you get when it’s inside? I believe this this only live inside th task
    jack

    jack

    3 months ago
    This pattern is repeated endlessly, requiring CTRL+C to break out
    File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1446, in info
        self._log(INFO, msg, args, **kwargs)
      File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1589, in _log
        self.handle(record)
      File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1599, in handle
        self.callHandlers(record)
      File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
        hdlr.handle(record)
      File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 952, in handle
        self.emit(record)
      File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1086, in emit
        stream.write(msg + self.terminator)
      File "/Users/jd/.pyenv/versions/prefect-logs/lib/python3.9/site-packages/prefect/utilities/logging.py", line 400, in write
        <http://self.stdout_logger.info|self.stdout_logger.info>(s)
    Kevin Kho

    Kevin Kho

    3 months ago
    I do get some weirdness, but not the endlessness
    jack

    jack

    3 months ago
    I found a pattern that works for what I need, and pasted it here.
    Kevin Kho

    Kevin Kho

    3 months ago
    Ah nice! Thanks for that!