Redirecting stdout from multiple processes to python Logging module

Question

I have a python script that launches a number of user processes using subprocess.Popen. Each process stdout is redirected to a unique file. For example the I launch each process as follows

proc = my_proc  
for p in range(1, max_p, 1):  
    log_file = proc + "_" + str(p) + ".log"  
    log = open(log_file, "w+")  
    subprocess.Popen([my_proc, p], shell = False, stdout = log)

I would like to rotate these files when they become too big. What would the best way of doing this? I would like to use the logging module but I dont think this is its intended use

Thanks

dr jimbob · Accepted Answer · 2010-11-30 16:49:04Z

1

Not a pythonic solution; but on linux systems I prefer using logrotate to automatically rotate my logs. Check to see if its installed on your system (On ubuntu say there is a directory called /etc/logrotate.d/ with files automatically run via cron). This may or may not be preferred to having log rotation run from within the application.

It's very configurable, e.g., allows compression of older files keeps N files via rotate N command, rotates when the cron over "size 100k", and looking at man logrotate, its very straightforward to setup.

From the man page here's a sample file

   # sample logrotate configuration file
   compress

   /var/log/messages {
       rotate 5
       weekly
       postrotate
           /usr/bin/killall -HUP syslogd
       endscript
   }

   "/var/log/httpd/access.log" /var/log/httpd/error.log {
       rotate 5
       mail [email protected]
       size 100k
       sharedscripts
       postrotate
           /usr/bin/killall -HUP httpd
       endscript
   }

answered Nov 30, 2010 at 16:49

dr jimbob

17.8k7 gold badges63 silver badges84 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

mikip Over a year ago

Thanks, however ther may be upto 100 processes launched from the python script. Would I need an entry for each process and logfile ?

dr jimbob Over a year ago

You do not need entries for every process/logfile -- it supports wildcards. E.g., if all your logs are stored in a directory /var/log/some_app/*.log can be used to rotate all the log files in that directory. This is actually the third example, that I cut off.

Johan Kotlinski · Accepted Answer · 2010-11-30 16:41:59Z

0

How about logging.handlers.RotatingFileHandler?

answered Nov 30, 2010 at 16:41

Johan Kotlinski

25.8k10 gold badges80 silver badges102 bronze badges

2 Comments

mikip Over a year ago

Thanks, but I need a logger for each process. Is that possible with this class?

Johan Kotlinski Over a year ago

It should be possible if you have an id for each process - then you can bake the id into the filename.

Collectives™ on Stack Overflow

Redirecting stdout from multiple processes to python Logging module

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related