Python logging and rotating files

Question

I have a python program that is writing to a log file that is being rotated by Linux's logrotate command. When this happens I need to signal my program to stop writing to the old file and start writing to the new one. I can handle the signal but how do I tell python to write to the new file?

I am opening the file like this:

logging.basicConfig(format='%(asctime)s:%(filename)s:%(levelname)s:%(message)s',filename=log_file, level=logging.INFO)

and writing to it like this:

logging.log(level,"%s" % (msg))

The logging modules look very powerful but also overwhelming. Thanks.

proski · Accepted Answer · 2015-02-04 23:28:38Z

47

Don't use logging.basicConfig, use WatchedFileHandler. Here's how to use it.

import time
import logging
import logging.handlers

def log_setup():
    log_handler = logging.handlers.WatchedFileHandler('my.log')
    formatter = logging.Formatter(
        '%(asctime)s program_name [%(process)d]: %(message)s',
        '%b %d %H:%M:%S')
    formatter.converter = time.gmtime  # if you want UTC time
    log_handler.setFormatter(formatter)
    logger = logging.getLogger()
    logger.addHandler(log_handler)
    logger.setLevel(logging.DEBUG)

log_setup()
logging.info('Hello, World!')
import os
os.rename('my.log', 'my.log-old')
logging.info('Hello, New World!')

answered Feb 4, 2015 at 23:28

proski

4,05932 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

proski Over a year ago

@TyHitzeman You mean RotatingFileHandler. RollingFileHandler is a Java thing.

Tyler Dane Over a year ago

This works great for the OP and other Unix users. FYI for Windows users: you can't use WatchedFileHandler according to the docs. Checkout RotatingFileHandler instead

Andrew Walker · Accepted Answer · 2012-02-02 03:43:15Z

35

You may want to look at WatchedFileHandler to implement this, or as an alternative, implement log rotation with RotatingFileHandler, both of which are in the logging.handlers module.

answered Feb 2, 2012 at 3:43

Andrew Walker

42.7k9 gold badges64 silver badges86 bronze badges

3 Comments

fredsnertz Over a year ago

Thanks! If I need to I can go this route but currently the file is already being rotated outside of my code. I was hoping there was a way to close the log and reopen it.

fredsnertz Over a year ago

Ah ... WatchedFileHandler ... now I get it.

user9074332 Over a year ago

is there an option to have the rolled logs not currently being written to compressed (zipped)? E.g the docs mention that RotatingFileHandler would produce app.log.1, app.log.2, ... but ideally these rolled logs would be optionally compressed to save space.

avasal · Accepted Answer · 2012-02-02 04:15:58Z

13

from logging import handlers

handler = handlers.TimedRotatingFileHandler(filename, when=LOG_ROTATE)

handler.setFormatter(logging.Formatter(log_format, datefmt="%d-%m-%Y %H:%M:%S"))

#LOG_ROTATE = midnight    
#set your log format

This should help you in handling rotating log

answered Feb 2, 2012 at 4:15

avasal

14.9k4 gold badges33 silver badges49 bronze badges

Comments

Alastair Irvine · Accepted Answer · 2013-12-24 05:42:03Z

9

Since rotation is already being done by logrotate, in your signal handler you should just call logging.basicConfig(...) again and that should reopen the log file.

answered Dec 24, 2013 at 5:42

Alastair Irvine

1,17812 silver badges16 bronze badges

1 Comment

Edmund Over a year ago

basicConfig is a no-op if it's already been run. To force it to reconfigure everything again, you have to set logging.root.handlers = [] first.

Collectives™ on Stack Overflow

Python logging and rotating files

4 Answers 4

2 Comments

3 Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

3 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related