slow File Processing in python

Question

I am trying a file operation using python.Aim is to continuously read a file of size(100bytes),pack and send them through socket. These files are read from a directory.

Problem: when i run the program continuously, execution time is increasing. Initially, execution time is less than a second; later it reaches till 8~10seconds. I am not able get exact reason for the delay.If anyone can throw some light on the issue, it will be more helpful.

Here i have attached my code...

def handlefile(filename):
        for sat in range(len(Numfiles)):
                filename = 
                fsize = os.path.getsize(filename)
                if fsize != 100:
                        continue
                rfile = open(filename,'rb')
                text = rfile.read()
                msg = struct.unpack("<100b",text)
                for i in range(len(msg)):
                        packMessage  = packMessage + struct.pack("<b",msg[i])
                print "time:",datetime.datetime.now() - startTime

The file are binary files.

Initial time taken : 671 ms

on executing continuously for more than 10 times,time increases slowly. Last few values, 671ms . . . . 9.879 ms 88.686 ms 135.954 ms

I am using python-2.5.4 version.

If anyone had come across similar problem. Please provide me some inputs.

Thanks Das

Please post actual running code, there are several syntax errors in this that would prevent it from running. — Daniel Roseman
– Daniel Roseman, Commented Oct 27, 2010 at 10:22

sastanin · Accepted Answer · 2010-10-27 10:34:05Z

4

From what I see, packMessage is growing monotonically:

packMessage  = packMessage + struct.pack("<b",msg[i])

If you repeat it many times, it may grow big, consume a lot of memory, and at some point your application may become much slower. Try looking at top or htop when you run your program (in top, press M to sort by memory allocation, f to add resident memory field).

Also opening and reading the same file every time is not the best solution from the performance point of view. Consider reading it only once before entering the loop.

answered Oct 27, 2010 at 10:34

sastanin

41.9k15 gold badges106 silver badges131 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Björn Pollex · Accepted Answer · 2010-10-27 10:10:33Z

3

Have you checked the number of file-handles your process has open? You may want yo use the with-statement to make sure they get closed when not needed anymore:

with open(filename, 'rb') as rfile:
    text = rfile.read()
    # etc.

When the with-block is left, the file will closed automatically.

answered Oct 27, 2010 at 10:10

Björn Pollex

77.1k30 gold badges206 silver badges290 bronze badges

Collectives™ on Stack Overflow

slow File Processing in python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related