Python - Logging from multiple modules to rotating files without printing to console

Question

I am trying to add logging to a medium size Python project with minimal disruption. I would like to log from multiple modules to rotating files silently (without printing messages to the terminal). I have tried to modify this example and I almost have the functionality I need except for one issue.

Here is how I am attempting to configure things in my main script:

import logging
import logging.handlers
import my_module

LOG_FILE = 'logs\\logging_example_new.out'

#logging.basicConfig(level=logging.DEBUG)

# Define a Handler which writes messages to rotating log files.
handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes=100000, backupCount=1)
handler.setLevel(logging.DEBUG)     # Set logging level.
# Create message formatter.
formatter = logging.Formatter('%(asctime)s %(name)-12s %(levelname)-8s %(message)s')
# Tell the handler to use this format
handler.setFormatter(formatter)

# Add the handler to the root logger
logging.getLogger('').addHandler(handler)

# Now, we can log to the root logger, or any other logger. First the root...
logging.debug('Root debug message.')
logging.info('Root info message.')
logging.warning('Root warning message.')
logging.error('Root error message.')
logging.critical('Root critical message.')

# Use a Logger in another module.
my_module.do_something()                     # Call function which logs a message.

Here is a sample of what I am trying to do in modules:

import logging

def do_something():
    logger = logging.getLogger(__name__)
    logger.debug('Module debug message.')
    logger.info('Module info message.')
    logger.warning('Module warning message.')
    logger.error('Module error message.')
    logger.critical('Module critical message.')

Now, here is my problem. I currently get the messages logged into rotating files silently. But I only get warning, error, and critical messages. Despite setting handler.setLevel(logging.DEBUG).

If I uncomment logging.basicConfig(level=logging.DEBUG), then I get all the messages in the log files but I also get the messages printed to the terminal.

How do I get all messages above the specified threshold to my log files without outputing them to the terminal?

Thanks!

Update: Based on this answer, it appears that calling logging.basicConfig(level=logging.DEBUG) automatically adds a StreamHandler to the Root logger and you can remove it. When I did remove it leaving only my RotatingFileHandler, messages no longer printed to the terminal. I am still wondering why I have to use logging.basicConfig(level=logging.DEBUG) to set the message level threshold, when I am setting handler.setLevel(logging.DEBUG). If anyone can shed a little more light on these issues it would still be appreciated.

To make sure the logging you're using is the logging you think you're using, do pip uninstall logging && pip install logging and now do python -c "import os, logging ; print os.path.dirname(logging.__file__)" to make sure. — boardrider
– boardrider, Commented Jun 10, 2015 at 21:14
if you use logging.basicConfig(level=logging.DEBUG) then you would see the output which is expected behaviour unless you supplied filename=... — Padraic Cunningham
– Padraic Cunningham, Commented Jun 10, 2015 at 21:14
@Padraic Cunningham, If I supply a file name to logging.basicConfig() then you are right, it does not print to the terminal. That is a step forward. It does however add a FileHandler to the Root logger and along with the RotatingFileHandler that I add. This means I get two copies of each message in my log files. I can remove the automatically created handler (please see my Update above) if that is what has to be done. Is there some way to sort out the message level issue in my log files without resorting to calling logging.basicConfig() ? Thanks. — user3692971
– user3692971, Commented Jun 10, 2015 at 21:58
If you want all the output logged for all levels in your code you should attach the handler with the highest level. — Padraic Cunningham
– Padraic Cunningham, Commented Jun 10, 2015 at 22:03

Community · Accepted Answer · 2017-05-23 11:58:33Z

5

You need to call set the logging level on the logger itself as well. I believe by default, the logging level on the logger is logging.WARNING

Ex.

root_logger = logging.getLogger('')
root_logger.setLevel(logging.DEBUG)

# Add the handler to the root logger
root_logger.addHandler(handler)

The loggers log level determines what the logger will actually log (i.e what messages will actually get handed to the handlers). The handlers log level determines what it will actually handle (i.e. what messages actually are output to file, stream, etc). So you could potentially have multiple handlers attached to a logger each handling a different log level.

Here's an SO answer that explains the way this works

edited May 23, 2017 at 11:58

CommunityBot

11 silver badge

answered Jun 10, 2015 at 22:05

junnytony

3,5551 gold badge25 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Kevin Glasson Over a year ago

I spent so long trying to figure out why my imported logging wasn't working, and it was because of this! I never grabbed the root logger with logging.getLogger('') i only had logging.getLogger(__name__) in my main()

Collectives™ on Stack Overflow

Python - Logging from multiple modules to rotating files without printing to console

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related