Custom method for python log file rotations?

Question

I am not able to get log file rotations to work in Python (v2.6.6). I need the naming control for log file extensions I find with TimedRotatingFileHandler, and the ability to force log file rotations at size thresholds I find with RotatingFileHandler.

If I establish a log handler fh of type TimedRotatingFileHandler I am able to include a DT stamp in the extension of the rotated log files. I do that like so:

 fh = logging.handlers.TimedRotatingFileHandler('/mydirpath/logs/logFileBaseName', when='H', interval=1, backupCount=48)
 fh.suffix = '%Y%m%d_%H%M%S'
 fh.extMatch = re.compile(r"^\d{8}\_\d{6}$")

However, apparently if my python script is not actively writing to the logs when interval occurs, the logs will not rotate. The single log file grows and grows to extreme sizes.

As an alternative I tried a simple RotatingFileHandler. It seems to give me the ability to dictate log rotation because it allows me to dictate a size threshold. Once that threshold is reached or even as it is approached, the logs rotate. This is how fh is established in my case for this handler type:

 fh = logging.handlers.RotatingFileHandler('/mydirpath/logs/logFileBaseName', maxBytes=7000000, backupCount=48)

Using RotatingFileHandler the logs are indeed rotated as expected, but it appears that I have no ability to add my DT stamp to the rotated log file extensions. As detailed in the documentation, I get extensions of .1, .2, .3, etc etc. I must have those DTs in the extensions of the rotated log files.

I suspect I need to use RotatingFileHandler but employ an override method to customize the extensions of the rotated log files. I'm not yet proficient at python, and do not know how I would identify the proper method in the logging package to override. I also do not know how I would establish such an override method in my python class to override a method in a package class.

I'm hoping to get some assistance. If I am misunderstanding or misusing the python rotating file handlers, I'd like to fix my error. If I am correct that a method override is required, I hope to get an example from someone in the community who has done something similar. Thanks in advance for any assistance.

stackoverflow.com/questions/8467978/… This is what you are looking for.. — gopalakrishnan
– gopalakrishnan, Commented Jul 6, 2018 at 7:03

Serge Ballesta · Accepted Answer · 2017-10-04 12:33:06Z

0

You really do not want that. The file name conventions are not accidental, but allow to make sure that your logging system behaves correctly:

when you rotate on a timely manner, you know in advance at what time the log files should be created, so using a timestamp in the name makes sense
when you rotate on a size threshold, you cannot know in advance at what time the log files should be created. If you name them with their creation timestamp, you cannot know whether a log file has been lost, so they get consecutive numbers in their names

That being said, all common file systems have the creation and modification time of the file as file properties, so you could build a python script that copy/move the files using their creation time in their new name. That way you do have not to clutter the Python logging system, and only need a single easy to write and test script using os.stat.

answered Oct 4, 2017 at 12:33

Serge Ballesta

150k13 gold badges137 silver badges267 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

56lt56 Over a year ago

I don't understand. The base file name in either case is not cretaed with a datetime stamp. The DT is embedded in the rotated log files at the time of their creation - why in either case would it matter if it is basename.1 or basename.20171004_101007? TimedRotatingFileHandler is able to track and expire my sequence of rotated log files as they age, because it can be told to look for all files that start with basename and end in ^\d{8}_\d{6}$ . These extensions are only added at the time the base log is rotated. What would it matter whether the rotation is time or size driven?

56lt56 Over a year ago

I guess it boils down to this: why doesn't this fh = logging.handlers.TimedRotatingFileHandler('/mydirpath/logs/logFileBaseName', when='H', interval=1, backupCount=48) give me a rotation of log output every hour?

Collectives™ on Stack Overflow

Custom method for python log file rotations?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related