How to limit log file size in python

Question

I am using windows 7 and python 2.7. I want to limit my log file size to 5MB. My app, when it starts, writes to log file, and then the app terminates. When my app starts again, it will write in same log file. So app is not continuously running. App initiates, processes and terminates.

My code for logging is:

import logging
import logging.handlers
logging.basicConfig(filename=logfile.log, level="info", format='%(asctime)s %(levelname)s %(funcName)s(%(lineno)d) %(message)s')
logging.info("*************************************************")

I tried with RotatingFileHandler but it didn't work

logging.handlers.RotatingFileHandler(logFile, mode='a', maxBytes=5*1024*1024, backupCount=2, encoding=None, delay=0)

So, how can I enforce a file size limit in python?

RotatingFileHandler is a way to go. How exactly it didn't work? — J0HN
– J0HN, Commented Jul 1, 2014 at 7:58
may be because app is not continuosly running or is there any wrong in coding — imp
– imp, Commented Jul 1, 2014 at 7:59
That doesn't answer my question :) I'm not asking you to speculate on why it's so, but WHAT exactly is wrong with RotatingFileHandler. Errors, exceptions, crashes, anything? Does it write to log at all? — J0HN
– J0HN, Commented Jul 1, 2014 at 8:06
@imp it doesn't matter how many times it gets interrupted. See getLogger. It will always return a reference to the same logger — Shadow9043
– Shadow9043, Commented Jul 1, 2014 at 8:19

betontalpfa · Accepted Answer · 2020-09-17 13:16:29Z

111

Lose basicConfig() and use RotatingFileHandler:

import logging
from logging.handlers import RotatingFileHandler

log_formatter = logging.Formatter('%(asctime)s %(levelname)s %(funcName)s(%(lineno)d) %(message)s')

logFile = 'C:\\Temp\\log'

my_handler = RotatingFileHandler(logFile, mode='a', maxBytes=5*1024*1024, 
                                 backupCount=2, encoding=None, delay=0)
my_handler.setFormatter(log_formatter)
my_handler.setLevel(logging.INFO)

app_log = logging.getLogger('root')
app_log.setLevel(logging.INFO)

app_log.addHandler(my_handler)

while True:
    app_log.info("data")

edited Sep 17, 2020 at 13:16

betontalpfa

3,8462 gold badges40 silver badges74 bronze badges

answered Jul 1, 2014 at 8:09

Shadow9043

2,2701 gold badge17 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

geosmart Over a year ago

just work in current py file ,how to make it work in other py files

Alex Over a year ago

@geosmart logging.getLogger('root').info('some log') in any python file should work

Vishal Rangras Over a year ago

For Python 3, don't forget to update the delay flag from 0 to False. It may appear like an integer but it is actually a boolean parameter.

Jaun Mauro Fernandez · Accepted Answer · 2019-12-15 13:54:09Z

19

When you use logging.basicConfig with a file, the log is attached with a file handler to handle writing to the file. afterwards you created another file handler to the same file with logging.handlers.RotatingFileHandler

Now, once a rotate is needed, RotatingFileHandler is trying to remove the old file but it can't becuase there is an open file handler

this can be seen if you look directly at the log file handlers -

import logging
from logging.handlers import RotatingFileHandler

log_name = 'c:\\log.log'
logging.basicConfig(filename=log_name)
log = logging.getLogger()
handler = RotatingFileHandler(log_name, maxBytes=1024, backupCount=1)
log.addHandler(handler)


[<logging.FileHandler object at 0x02AB9B50>, <logging.handlers.RotatingFileHandler object at 0x02AC1D90>]

edited Dec 15, 2019 at 13:54

Jaun Mauro Fernandez

32 bronze badges

answered Jun 16, 2015 at 16:09

AmitE

8941 gold badge9 silver badges13 bronze badges

Comments

ABL · Accepted Answer · 2020-08-05 14:47:19Z

18

To use BasicConfig and RotatingFileHandler, add RotatingFileHandler as Handler in BasicConfig.

main.py:

import logging

rfh = logging.handlers.RotatingFileHandler(
    filename='foo.log', 
    mode='a',
    maxBytes=5*1024*1024,
    backupCount=2,
    encoding=None,
    delay=0
)

logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s %(name)-25s %(levelname)-8s %(message)s",
    datefmt="%y-%m-%d %H:%M:%S",
    handlers=[
        rfh
    ]
)

logger = logging.getLogger('main')

logger.debug("test")

other.py

import logging

class Other():
    def __init(self):
        self.logger = logging.getLogger('other')
        self.logger.info("test2")

"test" will be written into foo.log with the tag 'main'

"test2" will be written into foo.log with the tag 'other'

answered Aug 5, 2020 at 14:47

ABL

1811 silver badge3 bronze badges

1 Comment

Nazar Over a year ago

No writes to foo.log happen for me (

Collectives™ on Stack Overflow

How to limit log file size in python

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related