4

I have very big logs (several gigabytes per day), that can (but do not need to) contain specific lines. I have to count the number of occurences of every one of these lines on a daily basis.

I have a file patterns.in, that contains the desired lines. For example:

aaaa
bbbb
cccc
dddd
eeee
ffff

The log files can look like this:

asd
dfg
aaaa
aaaa
sa
sdf
dddd
dddd
dddd
dddd
ghj
bbbb
cccc
cccc
cccc
fgg
fgh
hjk

The first (and perhaps most obvious approach) is to use grep, sort and uniq in the following way:

grep -f patterns.in logfile.txt | sort | uniq -c

which gives the following result:

   2 aaaa
   1 bbbb
   3 cccc
   4 dddd

It is close to what I want to achieve, but my desired result is:

   2 aaaa
   1 bbbb
   3 cccc
   4 dddd
   0 eeee
   0 ffff

So the problem is: how to print '0' if a line from pattern.in file is not matched? It needs to be done in a simplest possible way, as all I have available is the cygwin environment.

Improve this question
1
  • 2
    BTW, you might need grep's -o option otherwise multiple lines that, e.g., contain 'aaaa' plus other (different) data will be treated as non-unique by the sort | uniq -c Commented Oct 11, 2012 at 5:36

1 Answer 1

Reset to default
8

how about feeding the pattern file back in as a data file so that each pattern finds at least one match, and then subtracting one from the final reported count for each match

grep -f patterns.in logfile.txt patterns.in | cut -f2 -d':' | sort | uniq -c | awk '{print($1 - 1" "$2)}'
Improve this answer
4
  • Seems to be working well on examples, will need to check tommorow in production, as I am not sure if I have awk available there. Commented Oct 10, 2012 at 18:33
  • 2
    +1, nice answer. you can use grep's -h or --no-filename option to stop grep from printing filenames. e.g. grep -h -o -f patterns.in logfile.txt patterns.in sort | uniq -c | awk '{print($1 - 1" "$2)}' Commented Oct 11, 2012 at 5:37
  • It almost works, with one exception - awk '{print($1 - 1" "$2)}' prints only the first word of matched line (second field, $2). If a line has multiple words, how can I write its whole contents, that is from $2 to end of line? Commented Oct 11, 2012 at 9:52
  • 2
    @gorkypl, incorporated Craig Sander's grep -h suggestion and modified the command to work with multiple input words, here goes. grep -h -f patterns.in logfile.txt patterns.in | sort | uniq -c | tr -s ' ' |awk ' {count=$1 - 1; file_name=$0; sub($1, "", file_name);print(count" "file_name)}' Commented Oct 11, 2012 at 16:57

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.