Bash script optimisation

Question

This is the script in question:

for file in `ls products`
do
  echo -n `cat products/$file \
  | grep '<td>.*</td>' | grep -v 'img' | grep -v 'href' | grep -v 'input' \
  | head -1  | sed -e 's/^ *<td>//g' -e 's/<.*//g'`
done

I'm going to run it on 50000+ files, which would take about 12 hours with this script.

The algorithm is as follows:

Find only lines containing table cells (<td>) that do not contain any of 'img', 'href', or 'input'.
Select the first of them, then extract the data between the tags.

The usual bash text filters (sed, grep, awk, etc.) are available, as well as perl.

If you are not going to run this operation more than once or twice, who cares if it takes 1/2 a day to run? If you spend 2 hours optimizing it and only gain a 1 hour speed boost...is it worth it? — cdeszaq
– cdeszaq, Commented May 5, 2011 at 19:29
@cdeszaq: I have four other similar scripts that I believe I will be able to optimise once I've seen this one optimised. — Marko
– Marko, Commented May 5, 2011 at 19:34

glenn jackman · Accepted Answer · 2011-05-05 19:58:16Z

5

Looks like that can all be replace by one gawk command:

gawk '
    /<td>.*<\/td>/ && !(/img/ || /href/ || /input/) {
        sub(/^ *<td>/,""); sub(/<.*/,"")
        print
        nextfile
    }
' products/*

This uses the gawk extension nextfile.

If the wildcard expansion is too big, then

find products -type f -print | xargs gawk '...'

answered May 5, 2011 at 19:58

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

IanNorton · Accepted Answer · 2011-05-05 19:53:34Z

0

Here's some quick perl to do the whole thing that should be alot faster.

#!/usr/bin/perl

process_files($ARGV[0]); 

# process each file in the supplied directory
sub process_files($)
{
  my $dirpath = shift;
  my $dh;
  opendir($dh, $dirpath) or die "Cant readdir $dirpath. $!";
  # get a list of files
  my @files;
  do {
    @files = readdir($dh);
    foreach my $ent ( @files ){
      if ( -f "$dirpath/$ent" ){
        get_first_text_cell("$dirpath/$ent");
      }
    }
  } while ($#files > 0);
  closedir($dh);
}

# return the content of the first html table cell
# that does not contain img,href or input tags
sub get_first_text_cell($)
{
  my $filename = shift;
  my $fh;
  open($fh,"<$filename") or die "Cant open $filename. $!";
  my $found = 0;
  while ( ( my $line = <$fh> ) && ( $found == 0 ) ){
    ## capture html and text inside a table cell
    if ( $line =~ /<td>([&;\d\w\s"'<>]+)<\/td>/i ){
      my $cell = $1;

      ## omit anything with the following tags
      if ( $cell !~ /<(img|href|input)/ ){
        $found++;
        print "$cell\n";
      }
    }
  }
  close($fh);
}

Simply invoke it by passing the directory to be searched as the first argument:

$ perl parse.pl /html/documents/

edited May 5, 2011 at 19:53

answered May 5, 2011 at 19:41

IanNorton

7,3022 gold badges27 silver badges29 bronze badges

1 Comment

IanNorton Over a year ago

Running this on my system with a test set of 1000 files it takes less than a second.

hmontoliu · Accepted Answer · 2011-05-05 20:22:10Z

What about this (should be much faster and clearer):

for file in products/*; do
    grep -P -o '(?<=<td>).*(?=<\/td>)' $file | grep -vP -m 1 '(img|input|href)'
done

the for will look to every file in products. See the difference with your syntax.
the first grep will output just the text between <td> and </td> without those tags for every cell as long as each cell is in a single line.
finally the second grep will output just the first line (which is what I believe you wanted to achieve with that head -1) of those lines which doesn't contain img, href or input (and will exit right then reducing the overall time allowing to process the next file faster)

I would have loved to use just a single grep, but then the regex will be really awful. :-)

Disclaimer: of course I haven't tested it

Collectives™ on Stack Overflow

Bash script optimisation

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related