How can I decorate python logging output?

Question

I use import logging module for logging inside the AWS lambda with python 3.7 runtime.

I would like to perform certain manipulations on log statements before they are flushed to stdout, e.g. wrap the message as json and add tracing data, so that they would be parseable by Kibana parser.

I don't want to write my own decorator for that because that won't work for underlying dependencies.

Ideally, it should be something like a configured callback for the logger so that it would do following work for me:

    log_statement = {}
    log_statement['message'] = 'this is the message'
    log_statement['X-B3-TraceId'] = "76b85f5e32ce7b46"
    log_statement['level'] = 'INFO'
    sys.stdout.write(json.dumps(log_statement) + '\n')

while having still logger.info('this is the message').

How can I do that?

Configure the loghing with filters and formatters: docs.python.org/3/howto/logging.html — Klaus D.
– Klaus D., Commented Jun 11, 2020 at 14:40
This might provide a decent example: medium.com/analytics-vidhya/… — Yonah Dissen
– Yonah Dissen, Commented Jun 26, 2020 at 7:01

fyrkov · Accepted Answer · 2020-06-12 07:04:33Z

Answering my own question:

I had to use LoggerAdapter that is quite a good fit for the purpose of pre-processing log statements:

import logging

class CustomAdapter(logging.LoggerAdapter):

    def process(self, msg, kwargs):
        log_statement = '{"X-B3-TraceId":"%s", "message":"%s"}' % (self.extra['X-B3-TraceId'], msg) + '\n'
        return log_statement, kwargs

See: https://docs.python.org/3/howto/logging-cookbook.html#using-loggeradapters-to-impart-contextual-information

In general, the next step would be just plugging in the adapter like:

import logging
...
logging.basicConfig(format='%(message)s')
logger = logging.getLogger()
logger.setLevel(LOG_LEVEL)
custom_logger = CustomAdapter(logger, {'X-B3-TraceId': "test"})
...
custom_logger.info("test")

Note: I had to put format as a message only because I need to get the whole statement as a JSON string. Unfortunately, thus I lost some predefined log statement parts, e.g. aws_request_id. This is the limitation of LoggerAdapter#process as it handles only the message part. If anyone has a better approach here, pls suggest.

It appears that AWS lambda python runtime somehow interferes with logging facility and changing the format like above did not work. So I had to do additionally this:

FORMAT = "%(message)s"
logger = logging.getLogger()
for h in logger.handlers:
    h.setFormatter(logging.Formatter(FORMAT))

See: https://gist.github.com/niranjv/fb95e716151642e8ca553b0e38dd152e

Collectives™ on Stack Overflow

How can I decorate python logging output?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related