Logging to file and console with different handlers and formats

Question

I would like to have:

a main.log file with all logs above DEBUG level to be captured from main and imported modules
the console should show only ERROR level logs from main and its imported submodules.
- Note: I may have no control on the error handling logs of the imported submodules.

Here is the main.py code for this:

# main.py importing a submodule
import logging

import submodule

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# log to console
c_handler = logging.StreamHandler()

console_format = logging.Formatter("[%(levelname)s] %(message)s")
c_handler.setFormatter(console_format)
c_handler.setLevel = logging.INFO

logger.addHandler(c_handler)

logger.error("This is an error!!! Logged to console")

# log to file from main
logfile = "./logging/main.log"

f_handler = logging.FileHandler(filename=logfile)

f_format = logging.Formatter("%(asctime)s: %(name)-18s [%(levelname)-8s] %(message)s")
f_handler.setFormatter(f_format)
f_handler.setLevel = logging.DEBUG

logger.addHandler(f_handler)

logger.debug("This is a debug error. Not logged to console, but should log to file")

... and the submodule.py code ...

# submodule.py
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

formatter = logging.Formatter('%(levelname)s:%(name)s:%(message)s')

# log to console
c_handler = logging.StreamHandler()
c_handler.setFormatter(formatter)
logger.addHandler(c_handler)

logger.info("This is an info message from submodule, should be recorded in main.log!")
logger.debug("This is a debug message from submodule, also should be recorded in main.log!!")

When I run main.py:

[ERROR] This is an error!!! Logged to console shows up correctly in the console
But...
- Console also shows...
  - INFO:submodule:This is an info message from submodule, should be recorded in main.log!
  - [DEBUG] This is a debug error. Not logged to console, but should log to file
- The main.log file only shows yy-mm-dd hh:mm:ss: __main__ [DEBUG ] This is a debug error. Not logged to console, but should log to file only. It does not show logs from the submodule.py

Appreciate knowing:

Where am I going wrong?
What would be the code correction needed?

EDIT: Based on @Dan D. suggestion changed submodule.py as follows:

# submodule.py
import logging

logger = logging.getLogger(__name__)

def logsomething():
    logger.info("This is an info message from submodule, should be recorded in main.log!")
    logger.debug("This is a debug message from submodule, also should be recorded in main.log!!")

... and the program logs to console and file appropriately.

Q. If I want to change to message format for the submodule.py only, can this be done through main.py?

Dan D. · Accepted Answer · 2020-03-01 10:08:09Z

1

Your submodule should just be:

import logging

logger = logging.getLogger(__name__)

logger.info("This is an info message from submodule, should be recorded in main.log!")
logger.debug("This is a debug message from submodule, also should be recorded in main.log!!")

Then your main module should be:

# main.py importing a submodule
import logging

logger = logging.getLogger(__name__)

# log to console
c_handler = logging.StreamHandler()

console_format = logging.Formatter("[%(levelname)s] %(message)s")
c_handler.setFormatter(console_format)
c_handler.setLevel(logging.INFO)

logging.getLogger().addHandler(c_handler)

# log to file from main
logfile = "./logging/main.log"

f_handler = logging.FileHandler(filename=logfile)

f_format = logging.Formatter("%(asctime)s: %(name)-18s [%(levelname)-8s] %(message)s")
f_handler.setFormatter(f_format)
f_handler.setLevel(logging.DEBUG)

logging.getLogger().addHandler(f_handler)
logging.getLogger().setLevel(logging.DEBUG)

import submodule

logger.error("This is an error!!! Logged to console")
logger.debug("This is a debug error. Not logged to console, but should log to file")

Edit: The handlers have to be added before the code in the submodule runs. To effect this the import submodule was moved after the code that sets up the handlers.

Normally modules shouldn't have any top level logging calls so all the imports can be done at the top and then callables that use logging are called indirectly by the code in the if __name__=="__main__": after it sets up logging.

edited Mar 1, 2020 at 10:08

answered Mar 1, 2020 at 9:18

Dan D.

75k15 gold badges111 silver badges129 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

reservoirinvest Over a year ago

Changing the main and submodule as mentioned works well in console, but still does not log the submodule messages into main.log

reservoirinvest Over a year ago

Though it works, import submodule in the middle of main.py file doesn't seem to be best-practice, as VS Code setting: "editor.codeActionOnSave": {"source.organizeImports": true} puts it back on top. Would there be a better way?

Dan D. Over a year ago

You didn't read the second paragraph. It explains how to fix this by putting the code in the module in a function and then calling it from the main.

Collectives™ on Stack Overflow

Logging to file and console with different handlers and formats

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related