Explanation
Python official documentation (logging cookbook) suggests two approaches to add contextual information to logs:
- Using LoggerAdapters - for more details you can refer to pete lin`s answer.
- Using Filters (and global context variables) -
A filter processes the record before it is being emitted. It's main purpose is to allow advanced and customized rules to reject log records (the
filter method returns bool, which indices whether to emit the record). However, it also allows you to process the record - and add attributes based on whatever is required. For example, you can set the attributes based on a global context, for which you have two options:
- ContextVar for python 3.7 and above (thanks to jeromerg for his comment)
- threading.local variable for before python 3.7
When to use what?
- I'd recommend LoggerAdapters if you need to edit records for specific looger instances - in which case, it makes sense to instantiate an Adapter instead.
- I'd recommend Filter if you want to edit all records that are handled by a specific Handler - including other modules and 3rd party packages. It is generally a cleaner approach in my opinion, since we only configure our logger in our entry code - and the rest of the code remains the same (no need to replace logger instances with adapter instances).
Code Example
Below is a Filter example that appends attributes from a global threading.local variable:
log_utils.py
log_context_data = threading.local()
class ContextFilter(logging.Filter):
"""
This is a filter which injects contextual information from `threading.local` (log_context_data) into the log.
"""
def __init__(self, attributes: List[str]):
super().__init__()
self.attributes = attributes
def filter(self, record):
for a in self.attributes:
setattr(record, a, getattr(log_context_data, a, 'default_value'))
return True
And a ContextVar variant:
log_context_data = contextvars.ContextVar('log_context_data', default=dict())
class ContextFilter(logging.Filter):
"""
This is a filter which injects contextual information from `contextvars.ContextVar` (log_context_data) into the log.
"""
def __init__(self, attributes: List[str]):
super().__init__()
self.attributes = attributes
def filter(self, record):
context_dict = log_context_data.get()
for a in self.attributes:
setattr(record, a, context_dict.get(a, 'default_value'))
return True
log_context_data can be set when you start processing an account, and reset when you you're done. However, I recommend setting it using a context manager:
Also in log_utils.py for either threading.local:
class SessionContext(object):
def __init__(self, logger, context: dict = None):
self.logger = logger
self.context: dict = context
def __enter__(self):
for key, val in self.context.items():
setattr(log_context_data, key, val)
return self
def __exit__(self, et, ev, tb):
for key in self.context.keys():
delattr(log_context_data, key)
Or ContextVar
class SessionContext(object):
def __init__(self, logger, context: dict = None):
self.logger = logger
self.context: dict = context
self.token = None
def __enter__(self):
context_dict = log_context_data.get()
for key, val in self.context.items():
context_dict[key] = val
self.token = log_context_data.set(context_dict)
return self
def __exit__(self, et, ev, tb):
log_context_data.reset(self.token)
self.token = None
And a usage example, my_script.py:
root_logger = logging.getLogger()
handler = ...
handler.setFormatter(
logging.Formatter('{name}: {levelname} {account} - {message}', style='{'))
handler.addFilter(ContextFilter(['account']))
root_logger.addHandler(handler)
...
...
using SessionContext(logger=root_logger, context={'account': account}):
...
...
Notes:
Filter is only applied to the logger it is attached to. So if we attach it to a logging.getLogger('foo'), it won't affect logging.getLogger('foo.bar'). The solution is to attach the Filter to a Handler, rather than a logger.ContextFilter could've rejected records, if log_context_data doesn't contain the required attribute. This depends on what you need.
