Python: 3.7 json_logging: 1.3.0
I have a need to implement JSON-format logger for a Python application (for integration with Logstash). Currently, I have two different handlers implemented (one logging.StreamHandler for logging to console in a more "human" readable format, one logging.handlers.RotatingFileHandler).
I would like to retain the possibility of logging with these different handlers, as having JSON formatted output on console is not very friendly. I am using json_logging library for formatting the JSON logging part.
The issue I am having is trying to setup a custom JSON formatter. The default one has a lot of redundant information which I would like to remove (example, because I have custom logging function which create a log message from few input parameters, JSON formatted parameters "module" and "line_no" will always carry the same information (of my custom log function) and not the actual line number and module).
Logging setup is read from a configuration yaml file:
---
version: 1
disable_existing_loggers: False
formatters:
console_formatter:
format: "%(asctime)s %(process)d %(thread)d %(levelname)s %(message)s"
datefmt: "%Y-%m-%dT%H:%M:%S%z"
file_formatter:
format: "%(asctime)s %(process)d %(thread)d %(levelname)s %(message)s"
datefmt: "%Y-%m-%dT%H:%M:%S"
class: json_logging.JSONLogFormatter
handlers:
console:
class: logging.StreamHandler
level: DEBUG
formatter: console_formatter
stream: ext://sys.stdout
# Default log file gets written into a file inside the project folder
debug_file_handler:
class: logging.handlers.RotatingFileHandler
level: DEBUG
formatter: file_formatter
filename: /var/log/runtime.log
maxBytes: 10485760 # 10MB
backupCount: 30
encoding: utf8
loggers:
my_module:
level: DEBUG
handlers: [console]
propagate: no
root:
level: DEBUG
handlers: [console, debug_file_handler]
Snippet of the setup_logging function which reads the parameter file and sets up the logging:
if path is None:
logging.warning("Log configuration file NOT found: " + ','.join(value, default_path, package_path))
logging.basicConfig(level=default_level, format=format)
else:
logging.debug("Log configuration file found: " + path)
with open(path, 'rt') as f:
config = yaml.safe_load(f.read())
logging.config.dictConfig(config)
Using the class: json_logging.JSONLogFormatter in one of the handlers I am able to log in JSON format in only file_formatter and retain the non-JSON structure in console.
The issue comes into picture when I try to setup a custom JSON formatter. I am using the example provided in library's github: https://github.com/bobbui/json-logging-python/blob/master/example/custom_log_format.py
There, the custom class is provided for the init call to setup function json_logging.init_non_web(custom_formatter=CustomJSONLog, enable_json=True)
The problem is, that this function configures all handlers to use the same format. Which also changes the console formatter output to JSON.
I have tried to use getLogger and then setting the formatter to only the fileHandler, however that still changes all formatters:
log = logging.getLogger()
for hdlr in log.handlers[:]: # remove the existing file handlers
if isinstance(hdlr, logging.FileHandler):
hdlr.setFormatter(json_logging.init_non_web(custom_formatter=CustomJSONLog, enable_json=True))
Could someone help me in finding the way to use a custom JSON formatter for only a single handler?
