How to log errors in python when using the os module

Question

I'm trying to incorporate a simple way to keep track of a periodic mysqldump command I want to run using the os module in python. I've written this, but in testing it doesn't raise the exception, even when the mysqldump command completes with an error. I'm pretty new to python, so I might be approaching this terribly, but I thought I would try to get pointed in the right direction.

db_dump = "mysqldump -u %s -p%s --socket=source_socket --databases %s | mysql -u %s -p%s   --socket=dest_socket" % (db_user, db_pass, ' '.join(db_list), db_user, db_pass)

try:
    os.system(db_dump)
except:
    logging.error("databases did not dump")
else:    
    logging.info("database dump complete")

write loggings output to a file Using the logging python class to write to a file? — user1006989
– user1006989, Commented Dec 6, 2012 at 22:12

Cameron Sparr · Accepted Answer · 2012-12-06 22:12:40Z

5

os.system is not a very robust or powerful way to call system commands, I'd recommend using subprocess.check_output() or subprocess.check_call

ie,

>>> cmd = 'ls -l'
>>> badcmd = 'ls /foobar'
>>> subprocess.check_call(cmd.split())
0
>>> subprocess.check_call(badcmd.split())
ls: /foobar: No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ls', '/foobar']' returned non-zero exit status 1

answered Dec 6, 2012 at 22:12

Cameron Sparr

3,9912 gold badges26 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

numb3rs1x Over a year ago

Thank you for the tip. I will research subprocess and try to convert my code to use it once I'm comfortable with it.

abarnert Over a year ago

+1, but you shouldn't do cmd.split(). Just pass cmd as a string, and subprocess will break it up in a platform-appropriate way, taking into account quotes. (Of course it's better to have a list of args than a string in the first place—but if you do have a string, do not call split just to get a list of args.)

abarnert Over a year ago

@numb3rs1x: It really doesn't take much to convert your code—just replace every use of os.system with subprocess.check_call. You could even just search-and-replace this. It won't exactly the same thing os.system did—instead, it will do what you expected os.system to do, which is better. (If you do need anything fancy that os.system did, like passing shell expressions in the command, see the shell=True flag. But you probably don't.)

Cameron Sparr Over a year ago

@abamert: I believe you need to split() the command unless you call it with shell=True

numb3rs1x Over a year ago

@abamert I've been playing around with the subprocess module, and it does seem to be the way to go for running a shell. I'm struggling with how to handle the exceptions is raises so they go into a log file of my choosing.

steveha · Accepted Answer · 2012-12-06 22:18:07Z

2

os.system() returns an integer result code. When it returns 0, the command ran successfully; when it returns a nonzero value, that indicates an error.

db_dump = "mysqldump -u %s -p%s --socket=source_socket --databases %s | mysql -u %s -p%s   --socket=dest_socket" % (db_user, db_pass, ' '.join(db_list), db_user, db_pass)

result = os.system(db_dump)
if 0 == result:
    logging.info("database dump complete")
else:
    logging.error("databases did not dump; result code: %d" % result)

Like @COpython, I recommend the use of subprocess. It is a bit more complicated than os.system() but it is tremendously more flexible. With os.system() the output is sent to the terminal, but with subprocess you can collect the output so you can search it for error messages or whatever. Or you can just discard the output.

edited Dec 6, 2012 at 22:18

answered Dec 6, 2012 at 22:11

steveha

77.1k21 gold badges94 silver badges119 bronze badges

2 Comments

numb3rs1x Over a year ago

Thank you steveha for the quick and friendly response. I tried this and it worked, but I will research subprocess to see if that would be a better solution ultimately for this code.

steveha Over a year ago

With os.system() you get the exit status from the program you run. You can look at the "man page" for the program to see what error it returns. With subprocess you can get the exit status as well, but with os.system() that's all you get; you don't get an exception.

rh0dium · Accepted Answer · 2012-12-06 22:27:43Z

2

Here is what I would do.

import logging
import subprocess
log = logging.getLogger(__name__)

cmd = "mysqldump -u %s -p%s --socket=source_socket --databases %s | mysql -u %s -p%s " \
      "--socket=dest_socket" % (db_user, db_pass, ' '.join(db_list), db_user, db_pass)

process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE.PIPE)
stdout, stderr = process.communicate()
stdout = [x for x in stdout.split("\n") if x != ""]
stderr = [x for x in stderr.split("\n") if x != ""]

if process.returncode < 0 or len(stderr):
    for error in stderr:
        log.error(error)

answered Dec 6, 2012 at 22:27

rh0dium

7,0924 gold badges51 silver badges81 bronze badges

2 Comments

numb3rs1x Over a year ago

This looks a lot more advanced than I'm prepared for at my novice python level. I do thank you for contributing, and I'm looking forward to the day I when I have a better understanding of the intricacies of python programming.

rh0dium Over a year ago

Actually this isn't so bad. All we do is shove your command into subprocess (which is a better os.system - see stackoverflow.com/questions/4813238/…). Once we do that we wait for the results (communicate). Then split up the resulting lines into lines again, verify we got a good return and if not dump the output.

Collectives™ on Stack Overflow

How to log errors in python when using the os module

3 Answers 3

5 Comments

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related