python parse large logs to extract certain patterns

Question

I have large system log (max 1GB) and I need to parse it for extracting certain things.

initially i wrote python code to reading file line by line (using with open) it took very very long time

I learned about mmap and using mmap it's taking around 5 mins .

(I have precompiled the regular expression to save time)

is there any better approach so that it takes less time ? (i am using python 2.7.3 32bit )

axelcdv · Accepted Answer · 2014-09-06 10:28:42Z

1

You might want to use awk for that kind of things, it's more or less what it's designed to do.

answered Sep 6, 2014 at 10:28

axelcdv

7536 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1