a

    Alex Papanicolaou

    1 year ago
    Hey all, has anyone looked into using a progress bar like
    tqdm
    inside a task? I know there would be issues around stdout and how the prefect logger works but maybe someone has thought about this.
    nicholas

    nicholas

    1 year ago
    Hi @Alex Papanicolaou - I haven't tried this personally but since logging events aren't updatable I would imagine something like this wouldn't work with the Prefect Logger
    b

    Brett Naul

    1 year ago
    hey alex, we do this but it's not super easy this is what we use to do tqdm->logger
    class TqdmToLogger(StringIO):
        """Mock file handle to allow tqdm progress bar to write to logger.
    
        Adapted from <https://github.com/tqdm/tqdm/issues/313#issuecomment-267959111>.
        """
    
        def __init__(self, logger=None, level=<http://logging.INFO|logging.INFO>):
            self.buf = ""
            self.logger = logging.getLogger() if logger is None else logger
            self.level = level
            super().__init__()
    
        def write(self, buf):
            self.buf = buf.strip("\r\n\t ")
    
        def flush(self):
            self.logger.log(self.level, self.buf)
    
    
    class tqdm(_tqdm):
        """Wrapper around `tqdm.auto` that plays nicely with `fluentd` and Prefect logging."""
    
        def __init__(self, iterable=None, *args, file=None, **kwargs):
            if file is None and "notebook" not in _tqdm.__module__:
                file = TqdmToLogger()
            super().__init__(iterable, *args, file=file, **kwargs)
    we also have some magic that tries to make the default logger write to both prefect and stdout (not stderr in our case because we want the logs to show up in Stackdriver too) but I'm in the process of reworking that right now bc it doesn't always work...