Python logging fail to write into multiple log files

Question

I'd like to automate some tests and log the output of each test into different files, but it seems that all tests outputs are appended into the first log file, instead of separate files. May I know how can I correct this?

What I did is: First, define runthis.cfg which contains all the test config file. Second, execute run.py. It imports class A, for loop and write some log data for each loop

In runthis.cfg

2
prob1.cfg
prob2.cfg

In run.py

from fa import A

def run_it(cfg_data):
    cfg = cfg_data.split('\n')
    runCount = int(cfg[0])

    for i in range(1, runCount+1):        
        print "\n\nRunning " + cfg[i] + "..."
        cA.readProblemInstance(cfg[i])



cA = A() 

if __name__ == '__main__':    
    run_config_file = open('runthis.cfg', 'r')
    cfg_data = ''.join(run_config_file.readlines())
    run_config_file.close()        

    run_it(cfg_data)

In fa.py

from datetime import datetime
import time
import logging

class A():
    def activateLogFile(self, f):
        logging.basicConfig(filename=f, level=logging.INFO)

    def readProblemInstance(self, fn):    
        fn = fn.replace('.', '_')
        fn = fn + '_' + datetime.now().strftime('%Y_%m_%d_%H_%M_%S_%f')
        self.activateLogFile(fn)

        logging.info("%s" %fn)

The output is, in prob1_cfg_2014_04_07_12_39_38_293000,

INFO:root:prob1_cfg_2014_04_07_12_39_38_293000
INFO:root:prob2_cfg_2014_04_07_12_39_38_294000

prob2_cfg_2014_04_07_12_39_38_294000 doesn't exists!

metatoaster · Accepted Answer · 2014-04-07 03:57:03Z

2

logging.basicConfig doesn't replace already existing handlers (it does say, "This function does nothing if the root logger already has handlers configured for it."), which was set up by the very first call of that function. What you have to do is remove whatever handler attached to the root logger, and add a new one either by calling basicConfig again or other functions.

Note the following behavior, when you didn't remove the original handler

>>> rootlogger = logging.getLogger()
>>> logging.basicConfig(filename='/tmp/testlog.txt', level=logging.INFO)
>>> rootlogger.handlers
[<logging.FileHandler object at 0xd4a450>]
>>> logging.basicConfig(filename='/tmp/testlog.txt', level=logging.INFO)
>>> rootlogger.handlers
[<logging.FileHandler object at 0xd4a450>]

Note the same object id. Now assign the existing handler to a variable (persist it), then remove, then call logging.basicConfig again.

>>> handler = rootlogger.handlers[0]
>>> rootlogger.removeHandler(handler)
>>> logging.basicConfig(filename='/tmp/testlog.txt', level=logging.INFO)
>>> rootlogger.handlers
[<logging.FileHandler object at 0x1161350>]

Note the different object id.

edited Apr 7, 2014 at 3:57

answered Apr 7, 2014 at 3:00

metatoaster

19.2k5 gold badges65 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

twfx Over a year ago

may I know which header should I include for rootlogger?

ebarr Over a year ago

rootlogger isn't a package. It is the output of getLogger() with no arguments. rootlogger = logging.getLogger().

metatoaster Over a year ago

@ebarr thanks, I left out that line in my various copy/pasting. That's fixed now.

ebarr · Accepted Answer · 2014-04-07 04:38:56Z

1

@metatoaster has already given a good account of why your code does not log to multiple files, so this answer will only deal with how you log to multiple files given your example.

So the handlers for a a given logger are all stored in .handlers attribute of that logger. This is basically just a list that can be added to with the addHandler method of the logger. In this case your code would become:

# new style classes should inherit from object
class A(object):
    def __init__(self):
        # create a logger for the class and set the log level
        # here we use the root logger
        self.logger = logging.getLogger()
        self.logger.setLevel(logging.INFO)

    def activateLogFile(self, f):
        # create a handler with the name defined by the variable f
        handler = logging.FileHandler(f)
        # add that handler to the logger
        self.logger.addHandler(handler)

    def readProblemInstance(self, fn):    
        fn = fn.replace('.', '_')
        fn = fn + '_' + datetime.now().strftime('%Y_%m_%d_%H_%M_%S_%f')
        self.activateLogFile(fn)
        self.logger.info("%s" %fn)

answered Apr 7, 2014 at 4:38

ebarr

7,8521 gold badge32 silver badges43 bronze badges

4 Comments

twfx Over a year ago

Adding towhat @metatoaster has mentioned, I have added a few lines in your reply. It works correctly now, separate log in different files. Without adding this line, new log will be written in both older files and newly created log file.

ebarr Over a year ago

I was under the impression that the OP wanted to be able to log to two files at the same time. In that case there is no need to remove the previous handler.

twfx Over a year ago

alright, thanks anyway, I got the clues to solve this problem!

ebarr Over a year ago

Haha. I didn't realise you were the OP when I responded to that last comment. Fail. Yes if you want to update the handler then you can just cycle through the current handlers and remove them all, then add a new one. (I think you already suggested this in an edit)

Collectives™ on Stack Overflow

Python logging fail to write into multiple log files

2 Answers 2

3 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related