https://prefect.io logo
j

jack

06/17/2022, 5:23 PM
What would it take to have each task send logs both to the cloud and to a file on the local disk (A different file for each task)?
Here is a snippet that attaches two log handlers to the root logger. Note
configure_logging()
is called outside of the task. If
configure_logging()
is moved inside the task, exceptions are raised.
Copy code
import logging
import sys

import prefect
from prefect import Flow, task


def configure_logging():
    #logger = prefect.utilities.logging.get_logger()
    logger = logging.getLogger()

    # If your logger is the root logger, you may want to set it to INFO or DEBUG
    logger.setLevel('INFO')

    handler = logging.FileHandler('mylog.log', encoding='utf-8')
    handler2 = logging.StreamHandler(sys.stdout)

    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s',
                                  datefmt='%Y-%m-%d_%H:%M:%S')

    handler.setFormatter(formatter)
    handler2.setFormatter(formatter)

    logger.addHandler(handler)
    logger.addHandler(handler2)

# configure logging outside of task
configure_logging()

@task(log_stdout=True)
def log_something():
    <http://logging.info|logging.info>('Hello')

with Flow('log-test') as flow:
    log_something()


flow.run()
k

Kevin Kho

06/17/2022, 5:28 PM
What error do you get when it’s inside? I believe this this only live inside th task
j

jack

06/17/2022, 5:38 PM
This pattern is repeated endlessly, requiring CTRL+C to break out
Copy code
File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1446, in info
    self._log(INFO, msg, args, **kwargs)
  File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/Users/jd/.pyenv/versions/3.9.9/lib/python3.9/logging/__init__.py", line 1086, in emit
    stream.write(msg + self.terminator)
  File "/Users/jd/.pyenv/versions/prefect-logs/lib/python3.9/site-packages/prefect/utilities/logging.py", line 400, in write
    <http://self.stdout_logger.info|self.stdout_logger.info>(s)
k

Kevin Kho

06/17/2022, 6:03 PM
I do get some weirdness, but not the endlessness
j

jack

06/17/2022, 6:51 PM
I found a pattern that works for what I need, and pasted it here.
k

Kevin Kho

06/17/2022, 6:53 PM
Ah nice! Thanks for that!
3 Views