python logging: Custom Python LogRecord Throwing an error

Question

I am trying to use the python logging module to create a custom log file that records other information like host name and adds it to my DB. Below are the classes I created to do this, and the Handler part was working just fine, but now that I added a custom LogRecord class, it throws this error:

/src/lib/__init__.py", line 31, in __init__
logging.LogRecord.__init__(self, *args, **kwargs)
exceptions.TypeError: __init__() takes at most 9 arguments (10 given)

And here is how I execute it

logging.setLoggerClass(MyLogger)

log = logging.getLogger('testing')
log.addHandler(MyLogHandler())
d = {'host': '192.168.0.1'}
log.warn('Hi', d)

And here are the classes. It obviously has to do with the *args, **kwargs, but when I look at it, the *args is empty, and **kwargs only contains the d variable specified above. I don't understand the problem.

class MyLogRecord(logging.LogRecord):
    def __init__(self, *args, **kwargs):
        logging.LogRecord.__init__(self, *args, **kwargs) //THIS IS THE LINE IT DIES ON
        self.host = 'localhost'

class MyLogFormatter(logging.Formatter):

    def __init__(self, fmt, datefmt=None, host=None):
        logging.Formatter.__init__(self, fmt, datefmt)
        self.host = host

    def format(self, record):
        return logging.Formatter.format(record)

class MyLogger(logging.getLoggerClass()):
    def makeRecord(self, *args, **kwargs):
        return MyLogRecord(*args, **kwargs)

class MyLogHandler(logging.Handler): # Inherit from logging.Handler
    def __init__(self):
        # run the regular Handler __init__
        logging.Handler.__init__(self)
        # Our custom argument
        self.mongo = MongoLogger()

    def setupCustomLogger(self, name, this_host):
        formatter = MyLogFormatter(fmt='%(asctime)s - %(levelname)s - %(module)s - %(message)s - %(host)s')

        handler = logging.StreamHandler()
        handler.setFormatter(formatter)

        logger = logging.getLogger(name)
        logger.setLevel(logging.DEBUG)
        logger.addHandler(handler)
        return logger

    def emit(self, record):
        # record.message is the log message
        self.mongo.log(record)


class MongoLogger(object):

'''Logs messages to a MongoDB fh_admin log collection.'''
def log(self, message):
    #@todo write log to DB
    print message

You've not provided enough code to reproduce the problem. Even after removing the references to the classes that you haven't included (FirehoseLogRecord, MongoLogger) I cannot reproduce this error you're describing — Adam Wagner
– Adam Wagner, Commented Jun 30, 2012 at 3:08
Sorry I have updated the question. It shouldn't have been FirehoseLogRecord, it should have been MyLogRecord, which is what is causing the problem. — Nathan
– Nathan, Commented Jun 30, 2012 at 3:26
See, it dies on superclass init. Does plain logging.LogRecord.__init__ die on the same argument list? — 9000
– 9000, Commented Jun 30, 2012 at 3:50

Adam Wagner · Accepted Answer · 2012-06-30 16:44:25Z

10

The error is telling you exactly what's wrong; you are calling the constructor with too many arguments. To see what I mean, take a look at how log-records are ordinarily constructed in the default implementation of makeRecord:

def makeRecord(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None):
    """
    A factory method which can be overridden in subclasses to create
    specialized LogRecords.
    """
    rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
    if extra is not None:
        for key in extra:
            if (key in ["message", "asctime"]) or (key in rv.__dict__):
                raise KeyError("Attempt to overwrite %r in LogRecord" % key)
            rv.__dict__[key] = extra[key]
    return rv

Notice how makeRecord takes an extra param that it doesn't pass directly to LogRecord? You, on the other hand, are passing that directly to LogRecord.__init__, which is causing the error.

From here, you've got two options; you could provide a more complete implementation of makeRecord, or you could try using the LoggerAdapter class which should help you achieve the same goal with less code.

Here's an example:

# Common log info to be added to all logs reported with `log_adapter`
context = {'host': 'localhost'}

log = logging.getLogger('testing')
log.addHandler(logging.StreamHandler())
d = {'host': '192.168.0.1'}

log_adapter = logging.LoggerAdapter(log, context)
log_adapter.warning('Hi', d)

If you need to calculate the value of 'host' (for example) each time something is logged, you could make context an instance of a class that looks like a dictionary. Like so:

class LogContext(object):

    def __getitem__(self, key):
        if key == 'host':
            return 'localhost'
        raise KeyError(key)

    def __iter__(self):
        return iter(['host'])

log_adapter = logging.LoggerAdapter(log, LogContext())
log_adapter.warning('Hi', d)

One thing to note about LoggingAdapter, it apparently doesn't define all of the handy shortcut functions as the ordinary Logger class. That's why I've called the warning method instead of warn as you did above.

More info on LoggingAdapter and adding context to your logs can be found in the python docs.

NOTE - I didn't include MyLogHandler, MyLogFormatter, or MongoLogger in my examples as they were not relevant to the issue/error.

edited Jun 30, 2012 at 16:44

answered Jun 30, 2012 at 4:11

Adam Wagner

16.2k7 gold badges54 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Nathan Over a year ago

This makes a lot more sense, I noticed after I went back to where I got info from to reconstruct this for myself, the code was posted 5 years ago. A bit has probably changed since then. :) code.activestate.com/recipes/… I think I should be able to piece all this together to write my custom logs to the DB. Thank you so much for the detailed explanation!

Nathan Over a year ago

The LoggingAdapter looks like the way to go, plus I spent so much time messing around with logs, I think it all blurred together, much clearer now!

Nathan Over a year ago

I am guessing the LoggerAdapter doesn't have a logging.getLogger('testing') to make it easy to find this throughout the application?

Adam Wagner Over a year ago

@Hallik Not that I'm aware of, but you could create a simple function that does something similar ex: def getLogAdapter(name): context = {}; logging.LoggerAdapter(logging.getLogger(name), context). Or you just instantiate this adapter in some module then import it wherever you need to use it, or (depending on hour your app is structured, just pass it around.

Nathan Over a year ago

Excellent point. I am going to play with that, and using a Filter as well. Thanks!

|

Collectives™ on Stack Overflow

python logging: Custom Python LogRecord Throwing an error

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related