Search for multiple strings in large files with Python

One way is to cheat and imitate grep from unix OS, try http://nedbatchelder.com/code/utilities/pygrep.py

import os

# Get the pygrep script.
if not os.path.exists('pygrep.py'):
    os.system("wget http://nedbatchelder.com/code/utilities/pygrep.py")
from pygrep import grep, Options

# Writes a test file.
text="""This is a text
somehow there are many foo bar in the world.
sometimes they are black sheep, 
sometimes they bar bar black sheep.
most times they foo foo here
and a foo foo there"""
with open('test.txt','w') as fout:
    fout.write(text)

# Here comes the query
queries = ['foo','bar']

opt = Options() # set options for grep.
with open('test.txt','r') as fin:
    for i in queries:
        grep(i, fin, opt)
print

You can improve your code in at least three ways (in descending order of importance):

• You don't break out of the inner for loop when both lines are found. This means that the scrip will iterate over the entire file, despite the fact that the two lines are found somewhere in the beginning of the file.
• If the regexp pattern is identical for all files, you should compile the regexp outside of your outer for loop. If they change from file to file, put them outside your inner for loop. As it stands now, a new regexp object is created at each iteration.

Note: This may not be the case, since the most recent patterns are cached. (but there are no good reasons not to do it)

• Additionally, you shouldn't open and close the output file at each iteration.

The code below addresses these issues:

import re, os, codecs

path = "./"
dir_lib = os.listdir(path)
w_pattern = re.compile("(Keyword1\:)\s(\[(.+?)\])")
d_pattern = re.compile("Keyword2\s(\[(.+?)\])")

with open('output.txt', 'w') as outputfile:
    for book in dir_lib:
        if not book.endswith('.bin'):
            continue
        filename = os.path.join(path, book)
        with codecs.open(filename, "r", "utf-8", errors="ignore") as text:
            w_found, d_found = False, False
            for lineout in text:
                w = w_pattern.search(lineout)
                d = d_pattern.search(lineout)
                if w:
                    lineout = w.group(3)
                    outputfile.write(lineout + ",")
                    w_found = True
                elif d:
                    lineout = d.group(2)
                    outputfile.write(lineout + ";")
                    d_found = True
                if w_found and d_found:
                    break

A few simplifications which may or may not be applicable:

• I assume that Keyword1 and Keyword2 both occur at the start of a line (so I can use re.match instead of re.search)
• I assume that Keyword1 will always occur before Keyword2 (so I can search for one, then the other = half as many calls):

and so:

import codecs
import glob
import re

START = re.compile("Keyword1\:\s\[(.+?)\]").match
END   = re.compile("Keyword2\:\s\[(.+?)\]").match

def main():
    with open('output.txt', 'w') as outf:
        for fname in glob.glob('*.bin'):
            with codecs.open(fname, 'rb', 'utf-8', errors='ignore') as inf:
                w = None
                for line in inf:
                    w = START(line)
                    if w:
                        break

                d = None
                for line in inf:
                    d = END(line)
                    if d:
                        break

                if w and d:
                    outf.write('{0},{1};'.format(w.group(2), d.group(2)))

if __name__=="__main__":
    main()