How to write specific outputs to multiple log files using logging in a for loop in python?

Question

Goal: To run an experiment 'A' 10 times, each time with a different input. Experiment A returns a floating point value as its output (let us call it result). All the 10 results of the 10 experiments should be saved in a results file (using logging). There is a lot of logging that needs to be done while running experiment A (each time). All the intermediate outputs of the (10) experiments should be written to (10) different log files.

In other words, If my experiment directory is 'exp_dir'. There should exist a results (log) file after the end of the experiment that saves the output of each of the experiment (only the output needs to be saved, nothing else). During the process of running each experiment I will create sub directories (exp1, exp2 and so on) for each of the 10 experiments within the exp_dir and log the outcomes of the experiments in the corresponding sub directories (experiment 1 outcome should be saved in exp1 sub directory). I am planning to do this in a single python file using for loop.

Could you let me know how to do the logging separately? This is what I tried (instead of 10 experiments I tried on 2 experiments)

import logging
import os

class MyLogger(object):
    def __init__(self):
        self.logger = logging.getLogger()
        self.logger.setLevel(logging.INFO)

    def activateLogFile(self, f):
        d =  os.path.dirname(f)
        os.makedirs(d, exist_ok=True)
        handler = logging.FileHandler(f)
        self.logger.addHandler(handler)

    def doLogging(self, fn, info):
        self.activateLogFile(fn)
        self.logger.info("{i}\n".format(i=info))


def run():

    exp_dir = os.getcwd()
    myLog.doLogging("{d}/res.log".format(d=exp_dir), "This is where results should be logged")
    myLog.logger.propagate = False
    for i in range(1, 3):
        myLog.doLogging("{d}/exp{i}/info.log".format(i=i, d=exp_dir), "some logging")
    myLog.doLogging("{d}/res.log".format(d=exp_dir), "Experiments done")

myLog = MyLogger()

if __name__ == "__main__":
    run()

But everything that is logged in exp2 is also logged in exp1 and res.log consists of all the logging (exp1, exp2 also)

I appreciate your help

Just to make it clear, the question is how to do the logging of each experiment separately and also how to save the outcome of each experiment in a results file. — user1540393
– user1540393, Commented Aug 17, 2017 at 18:41
You've showed us what you want to do, but you haven't shown us what you tried. Stack Overflow is not a code writing service. You need to demonstrate effort on your part before posting a problem here. You've imported logging, but we don't see anywhere you've tried to do anything with it. Have you read the documentation? Tried something and it didn't work? Edit your question and post what you have tried. And don't comment on your question; edit the question itself and add your clarification. — AJNeufeld
– AJNeufeld, Commented Aug 17, 2017 at 18:55

AJNeufeld · Accepted Answer · 2017-08-17 20:53:59Z

Your code reads (approximately):

self.logger = logging.getLogger()

handler = logging.FileHandler(f1)
self.logger.addHandler(handler)
self.logger.info("Log exp 1 stuff")   # Logs to f1

handler = logging.FileHandler(f2)
self.logger.addHandler(handler)
self.logger.info("Log exp 2 stuff")   # Logs to f1, f2

handler = logging.FileHandler(f3)
self.logger.addHandler(handler)
self.logger.info("Log global stuff")   # Logs to f1, f2, f3

Note you are adding a second, and then a third handler to your log object. You are not removing the old handlers.

You could:

    def __init__(self):
        self.handler = None
        # ... other code ...

    def activateLogFile(self, f):
        # ... other code ...
        if self.handler is not None:
            self.logger.removeHandler(self.handler)
        self.handler = logging.FileHandler(f)
        self.logger.addHandler(self.handler)

As suggested in Removing handlers from python, perhaps you could simply use:

    def activateLogFile(self, f):
        # ... other code ...
        self.logger.handlers = []
        self.logger.addHandler(handler)

bergerg · Accepted Answer · 2017-08-17 21:31:37Z

0

What's wrong with keeping it simple?

import logging
from os.path import join, dirname, abspath

def fast_log(msg, filename='res.log'):
    logging.basicConfig(filename=filename, level=logging.INFO)
    logging.info(msg)

def do_experiment(i):
    filename = join(dirname(abspath(__file__)), "%d.log" % i)
    fast_log("experiment initialization log", filename)
    # do stuff
    fast_log("first stage", filename)
    # do a lot
    # more stuff
    # ...
    results = execute_final_stage_of_exp()
    return results

for i in range(1, 11):
    fast_log(do_experiment(i))

edited Aug 17, 2017 at 21:31

answered Aug 17, 2017 at 21:14

bergerg

9959 silver badges23 bronze badges

Comments

9dubs · Accepted Answer · 2023-12-27 07:10:20Z

0

okay i had the exact same problem. a new handler was added in every single iteration, so data was being logged in a recursive manner. say if i had 4 iterations and needed 10 entries each, my first log file had 40 entries, second had 30 and so on till last had 10.

to fix this i simply emptied the log handler at the end of every iteration, as suggested in one of the answers using

logger.handlers = []

answered Dec 27, 2023 at 7:10

9dubs

878 bronze badges

Collectives™ on Stack Overflow

How to write specific outputs to multiple log files using logging in a for loop in python?

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related