python subprocess read stdout as it executes

Question

one of my function in the program checks the md5sum of hashfile

def check():
    print "checking integrity status.."
    md5 = subprocess.Popen(["md5sum", "-c", hashfile],shell=False, stdout=subprocess.PIPE)
    #fopen = open(basefile, "r")
    for f in md5.stdout.readlines():
        fc = f.rstrip("\n")
        sys.stdout.write("\rChecking..." + fc)
        sys.stdout.flush()

now what happens is that the whole command is first executed and then for loop reads from md5 using md5.stdout.readlines, as such it is not dynamic i.e. i dont get the output as the command is executed....is there a way i can get the output while the command is in execution...

fixed using glglgl's answer:

def check():
    print "checking integrity status.."
    md5 = subprocess.Popen(["md5sum", "-c", hashfile],shell=False, stdout=subprocess.PIPE)
    #fopen = open(basefile, "r")
    fc = "NULL"
    i = 0
    for f in iter(md5.stdout.readline, ''):
        k = fc
        fc = f.rstrip("\n")
        if "FAILED" in fc:
            print fc
        i = i + 1
        sys.stdout.write("\rChecking... "+ str(i)+ " " + fc + (' ' * (len(k) - len(fc))) )
        sys.stdout.flush()

Related, but not an answer to the question itself: have a look at hashlib.md5(). Might be simpler? — Steven
– Steven, Commented Sep 8, 2011 at 20:41

glglgl · Accepted Answer · 2011-09-08 10:31:31Z

Of course. There are several ways.

First, you could work with md5.stdout.read(), but there you would have to do line separation by yourself.
Then, you could operate with the file object md5.stdout as iterator. But there seems to be an issue with buffering, i. e. you don't get the results immediately.
And then there is a possibility to call md5.stdout.readline() repeatedly until it returns ''.

The third way is to be preferred in this case; I would suggest it to do like this:

...

for f in iter(md5.stdout.readline, ''):
    fc = f.rstrip("\n")
    sys.stdout.write("\rChecked " + fc)
    sys.stdout.flush()

I have also changed the output text, as there is only an output if che check is done already.

If that is not what you want, but rather really haveing every output captured separately, you should switch to point 1. But that makes it more complicated. I will think about a solution on indication that it is wanted.

There, one must consider the following points:

read() blocks, so one should read byte for byte (quite ugly).
There is the question what should be output and when should there be a intermitting output.

Kyle L · Accepted Answer · 2012-05-15 15:18:15Z

1

The original poster is correct that hashlib is not available in Python 2.4, however, the md5 library is available. Example workaround:

try:
        # Python 2.5 and up.
        import hashlib
        md5Hash = hashlib.md5

except ImportError:
        # Python 2.4 and below.
        import md5
        md5Hash = md5.new

somedata = 'foobar'
hashstring = md5Hash (somedata).hexdigest ()

answered May 15, 2012 at 15:18

Kyle L

111 bronze badge

Comments

Diego Navarro · Accepted Answer · 2011-09-08 10:32:28Z

0

What is the file size?

Popen creates a new child process to run the command. Maybe it finishes before you run the for loop.

You can check if the "subprocess" has finished looking returncode attribute (in your code: md5.returncode)

answered Sep 8, 2011 at 10:32

Diego Navarro

9,7323 gold badges31 silver badges33 bronze badges

Collectives™ on Stack Overflow

python subprocess read stdout as it executes

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related