python logging: multiple loggers error

Question

I have objects called Job which has it's own logger (each Job need to have a log file which is represented by logging.getLogger())

The problem is I create thousands of Jobs (~4000) and they all want to create a logger.

Traceback (most recent call last):
  File "/u/lib/btool/Job.py", line 151, in __init__
  File "/usr/lib/python2.7/logging/__init__.py", line 911, in __init__
  File "/usr/lib/python2.7/logging/__init__.py", line 936, in _open
IOError: [Errno 24] Too many open files: '/x/zooland/20160710-032802.log'

Is there way to deal with multiple loggers?

A different logger file for each object seems to be an overkill to me. Can you not log by class names or types or something? — masnun
– masnun, Commented Jul 10, 2016 at 7:37
each job has a directory of its own and log file needs to be there. can a single logger write to different log files? but that still doesnt address the issue of the need to open thousands of log files, correct? — ealeon
– ealeon, Commented Jul 10, 2016 at 7:39
I do not know if the default FileHandler keeps the file object open. If it does, then another idea would be write your custom file handler which opens the file, appends the message and then closes the file. — masnun
– masnun, Commented Jul 10, 2016 at 7:40
or maybe only open when you need to write to it. does logging leaves the log file open? — ealeon
– ealeon, Commented Jul 10, 2016 at 7:40
yes i used to have a custom file handler that does that. i wanted to try logging as it is more widely used. but i guess it cant? — ealeon
– ealeon, Commented Jul 10, 2016 at 7:41

masnun · Accepted Answer · 2016-07-10 07:51:09Z

1

Here's a custom file handler that stores the log message and then closes the file.

import logging


class MyFileHandler(logging.Handler):
    def __init__(self, filename):
        self.filename = filename
        super().__init__()

    def emit(self, record):
        log_text = self.format(record)
        try:
            fh = open(self.filename, "a")
            fh.write(log_text)
            fh.close()

            return True
        except:
            return False


logger = logging.getLogger("job")

handler = MyFileHandler("file-1")
logger.addHandler(handler)

logger.error("hola")

answered Jul 10, 2016 at 7:51

masnun

12k4 gold badges44 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

ealeon Over a year ago

super().__init__() TypeError: super() takes at least 1 argument (0 given)

ealeon Over a year ago

do i have to pass something to super().__init__() ?

masnun Over a year ago

I am using Python 3. Are you on Python 2? In that case please follow the correct syntax.

ealeon Over a year ago

not sure the difference but would it better to extend logging.FileHandler instead of logging.Handler?

masnun Over a year ago

If you extend FileHandler then you inherit the same behavior. It would have been more complex. For Python 2.7, use super(MyFileHandler, self).__init__()

Collectives™ on Stack Overflow

python logging: multiple loggers error

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related