How to quicky create array from big file?

Question

I have example:

    for line in IN.readlines():
        line = line.rstrip('\n')
        mas = line.split('\t')
        row = ( int(mas[0]), int(mas[1]), mas[2], mas[3], mas[4] )
        self.inetnums.append(row)
    IN.close()

If ffilesize == 120mb, script time = 10 sec. Can I decrease this time ?

You're reading a 120GB file into memory? How much memory does your machine have? — interjay
– interjay, Commented Apr 5, 2012 at 10:36

jamylak · Accepted Answer · 2012-04-05 10:38:08Z

4

Remove the readlines()

Just do

for line in IN:

Using readlines you are creating a list of all lines from the file and then accessing each one, which you don't need to do. Without it the for loop simply uses the generator which returns a line each time from the file.

edited Apr 5, 2012 at 10:38

answered Apr 5, 2012 at 10:32

jamylak

134k30 gold badges238 silver badges240 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Abhijit · Accepted Answer · 2012-04-05 14:51:50Z

2

You may gain some speed if you use a List Comprehension

inetnums=[(int(x) for x in line.rstrip('\n').split('\t')) for line in fin]

Here is the profile information with two different versions

>>> def foo2():
    fin.seek(0)
    inetnums=[]
    for line in fin:
        line = line.rstrip('\n')
        mas = line.split('\t')
        row = ( int(mas[0]), int(mas[1]), mas[2], mas[3])
        inetnums.append(row)


>>> def foo1():
    fin.seek(0)
    inetnums=[[int(x) for x in line.rstrip('\n').split('\t')] for line in fin]

>>> cProfile.run("foo1()")
         444 function calls in 0.004 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.003    0.003    0.004    0.004 <pyshell#362>:1(foo1)
        1    0.000    0.000    0.004    0.004 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
      220    0.000    0.000    0.000    0.000 {method 'rstrip' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'seek' of 'file' objects}
      220    0.000    0.000    0.000    0.000 {method 'split' of 'str' objects}


>>> cProfile.run("foo2()")
         664 function calls in 0.006 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.005    0.005    0.006    0.006 <pyshell#360>:1(foo2)
        1    0.000    0.000    0.006    0.006 <string>:1(<module>)
      220    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
      220    0.001    0.000    0.001    0.000 {method 'rstrip' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'seek' of 'file' objects}
      220    0.001    0.000    0.001    0.000 {method 'split' of 'str' objects}


>>>

edited Apr 5, 2012 at 14:51

answered Apr 5, 2012 at 10:56

Abhijit

64k20 gold badges143 silver badges209 bronze badges

2 Comments

jamylak Over a year ago

Would you actually gain some speed by using list comps, other than the speed gained by removing readlines? It seems to me like its just another way of writing the same code.

Abhijit Over a year ago

@jamylak: Consider the fact that you won't be calling append multiple times in a loop. I have updated my answer with information from cProfile.

Collectives™ on Stack Overflow

How to quicky create array from big file?

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related