Setting up Python logging manually with MemoryHandler and FileHandler

Question

I can't find enough documentation to get this to work. I have non-longrunning processes that show progress information on stderr and use stdout for output. What I'd like is the log messages to appear all collated at the end when the processes exit, plus it should log to a file as well. From what I can find in the documentation, MemoryHandler and FileHandler are what I need. But when I set this up like below, I don't get any output, neither in the file nor on exit. Any help would be appreciated.

import logging, logging.handlers, atexit, sys

filename = 'mylogfile.txt'
logLevel = logging.DEBUG

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
streamhandler = logging.StreamHandler(sys.stderr)
streamhandler.setLevel(logLevel)
streamhandler.setFormatter(formatter)
memoryhandler = logging.handlers.MemoryHandler(1024*100, logLevel, streamhandler)

filehandler = logging.FileHandler(filename)
filehandler.setLevel(logLevel)
filehandler.setFormatter(formatter)

logger = logging.getLogger()
logger.addHandler(memoryhandler)
logger.addHandler(filehandler)
def flush():
    memoryhandler.flush()
atexit.register(flush)
logger.debug("Logger has Initialized")

Michel Müller · Accepted Answer · 2015-12-07 01:58:28Z

6

Vinay Sajip has provided the most important missing clue here: The logger indeed has its own level, which I forgot to set so DEBUG records would never be emitted. There was one more issue however, which lead to MemoryHandler to immediately flush after receiving a record: Its own level needs to be set higher for this to work, because it is not a lower cutoff level as I assumed, but a level at or above the handler immediately flushes its records to target. See also the docs for reference. The following code has fixed these problems and is written in a more self documenting way in order to be aware of this subtlety.

import logging, logging.handlers, atexit, sys

filename = 'mylogfile.txt'
logLevel = logging.DEBUG

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
streamhandler = logging.StreamHandler(sys.stderr)
streamhandler.setLevel(logLevel)
streamhandler.setFormatter(formatter)
memoryhandler = logging.handlers.MemoryHandler(
    capacity=1024*100,
    flushLevel=logging.ERROR,
    target=streamhandler
)

filehandler = logging.FileHandler(filename)
filehandler.setLevel(logLevel)
filehandler.setFormatter(formatter)
logger = logging.getLogger()
logger.setLevel(logLevel)
logger.addHandler(memoryhandler)
logger.addHandler(filehandler)
def flush():
    memoryhandler.flush()
atexit.register(flush)
logging.debug("Logger has Initialized")
sys.stderr.write("I'd like this printed on the console first\n")

stderr output:

I'd like this printed on the console first
2015-12-07 10:42:45,707 - DEBUG - Logger has Initialized

mylogfile.txt:

2015-12-07 10:42:45,707 - DEBUG - Logger has Initialized

edited Dec 7, 2015 at 1:58

answered Dec 7, 2015 at 1:50

Michel Müller

5,7353 gold badges35 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Vinay Sajip Over a year ago

You don't strictly need to specify flushLevel in this case, as the default is logging.ERROR if not specified.

Michel Müller Over a year ago

Thanks. IMO since the default behavior is a bit unexpected, stating it explicitely is a bit better in the sense of self documentation.

Saket Suraj Over a year ago

Hi Michel, I have some questions. Why the logger is flushing here? You have added just one record so, the buffer still have more memory right? Also the capacity is equal to the number of records so, why have you taken the capacity input in bytes? My questions might not sound good but I need the clarity so please answer.

Michel Müller Over a year ago

Hey @SaketSuraj. As far as I remember the flushing here is just triggered by the program exiting (atexit.register(flush)). Does that answer your question? edit: regarding capacity the docs are not explicit here, but as far as I remember it is specified in bytes.

Vinay Sajip · Accepted Answer · 2015-12-05 16:28:25Z

3

You need to specify a level on the logger as well as the handler. The handler's level is only consulted after the logger level is checked.

See here for more information.

answered Dec 5, 2015 at 16:28

Vinay Sajip

100k15 gold badges184 silver badges196 bronze badges

1 Comment

Michel Müller Over a year ago

Thank you @VinaySajip! If you don't mind, I've also created a more complete answer that fixed another issue I had, but your help is very much appreciated.

Collectives™ on Stack Overflow

Setting up Python logging manually with MemoryHandler and FileHandler

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related