1

I'm trying to read some files in a directory, which has 10 text files. With time, the number of files increases, and the total size as of now goes around 400MB.

File contents are in the format:

student_name:student_ID:date_of_join:anotherfield1:anotherfield2

In case of a match, I have to print out the whole line. Here's what I've tried.

findvalue = "student_id" #this is users input alphanumeric
directory = "./RecordFolder"
for filename in os.listdir(directory):
    with open(os.path.join(directory, filename)) as f:
        for line in f:
            if findvalue in line:
                print line

This works, but it takes a lot of time. How can I reduce the run time?

Improve this question
22
  • 2
    Define a lot of time :) In my case similar example takes about 0.392s on a 120M file. So should take about 1.2s on a 400M file. The only difference in your code and mine is that I'm explicitly opening file in 'r'ead-only mode. Commented Aug 4, 2012 at 15:51
  • 3
    One little hint, totally unrelated to your question: else:pass is totally unnecessary, lose it. if statements aren't required to have an else. Commented Aug 4, 2012 at 16:02
  • 1
    @Whiskey well, OS might make a difference, but not THAT much difference. I/O load on your HDD will, however, make a difference :) It's quite hard to say what's causing that delay. Basically your file read speed is as fast as your HDD allows you to read it. Commented Aug 4, 2012 at 16:12
  • 2
    @Whiskey that would be my safest bet without more information. As you saw in my paste example similar code traverses a 900M binary file in a bit under 4 seconds on an SSD powered machine. I do see a spike of 240MB/sec read during execution. Commented Aug 4, 2012 at 16:16
  • 2
    Outputting to a file instead of to the screen (assuming stdout here is a tty) may also speed things up significantly. Terminals are very slow to print things, comparatively. Commented Aug 4, 2012 at 21:19

2 Answers 2

Reset to default
4

When textfiles become too slow, you need to start looking at databases. One of the main purposes of databases is to intelligently handle IO from persistent data storage.

Depending on the needs of your application, SQLite may be a good fit. I suspect this is what you want, given that you don't seem to have a gargantuan data set. From there, it's just a matter of making database API calls and allowing SQLite to handle the lookups -- it does so much better than you!

If (for some strange reason) you really don't want to use a database, then consider further breaking up your data into a tree, if at all possible. For example, you could have a file for each letter of the alphabet in which you put student data. This should cut down on looping time since you're reducing the number of students per file. This is a quick hack, but I think you'll lose less hair if you go with a database.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

IO is notoriously slow compared to computation, and given that you are dealing with large files it's probably best deal with the files line by line. I don't see an obvious easy way to speed this up in Python.

Depending on how frequent your "hits" (i.e., findvalue in line) will be you may decide to write to a file so not to be possibly slowed down by console output, but if there will be relatively few items found, it wouldn't make much of a difference.

I think for Python there's nothing obvious and major you can do. You could always explore other tools (such as grep or databases ...) as alternative approaches.

PS: No need for the else:pass ..

Improve this answer

2 Comments

@downvoter A downvote without explanation doesn't help anyone (OP, SO or me) .. I'm always happy to correct errors or improve an answer, but that requires constructive feedback, i.e., more than an anonymous click.
You haven't really answered any of OP's question. You've just restated the problem. Your edits certainly make the answer more productive though, so I am removing my downvote.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.