is any of array items contained in a string

Question

I have a keywords list and a blacklist. I want to delete all keywords that contain any of blacklist item. At the moment Im doing it this way:

my @keywords = ( 'some good keyword', 'some other good keyword', 'some bad keyword');
my @blacklist = ( 'bad' );

A: for my $keyword ( @keywords ) {
    B: for my $bl ( @blacklist ) {
        next A if $keyword =~ /$bl/i;      # omitting $keyword
    }
    # some keyword cleaning (for instance: erasing non a-zA-Z0-9 characters, etc)
}

I was wondering is there any fastest way to do this, becouse at the moment I have about 25 milion keywords and couple of hundrets words in blacklist.

Do you want a new array with filtered @keywords?

mpapec
– mpapec

2013-05-24 09:21:25 +00:00
Commented May 24, 2013 at 9:21 — mpapec
– mpapec, Commented May 24, 2013 at 9:21
It can be a new array.

gib
– gib

2013-05-24 09:22:23 +00:00
Commented May 24, 2013 at 9:22 — gib
– gib, Commented May 24, 2013 at 9:22

Dave Sherohman · Accepted Answer · 2013-05-24 09:42:09Z

4

The most straightforward option is to join the blacklist entries into a single regular expression, then grep the keyword list for those which don't match that regex:

#!/usr/bin/env perl    

use strict;
use warnings;
use 5.010;

my @keywords = 
  ('some good keyword', 'some other good keyword', 'some bad keyword');
my @blacklist = ('bad');

my $re = join '|', @blacklist;
my @good = grep { $_ !~ /$re/ } @keywords;

say join "\n", @good;

Output:

some good keyword
some other good keyword

answered May 24, 2013 at 9:42

Dave Sherohman

46.4k14 gold badges67 silver badges104 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

gib Over a year ago

Thanks a lot! For a test with 50k keywords, execution time went down from 34sec to 0,6sec

Oesor Over a year ago

metacpan.org/module/Regexp::Assemble - Regexp::Assemble improves performance more.

Oesor Over a year ago

To demonstrate: perl -MData::Printer -MRegexp::Assemble -E "my $ra = Regexp::Assemble->new(); for my $word (qw/apple asp application aspire applicate aardvark snake/) { $ra->add($word) } p($ra->re);" gives (?:a(?:ppl(?:icat(?:ion|e)|e)|sp(?:ire)?|ardvark)|snake)

AdrianHHH · Accepted Answer · 2013-05-24 09:39:51Z

3

Precompiling the search may help my @blacklist = ( qr/bad/i ) if you want to keep the nested loops.

Alternatively, changing from my @blacklist = ( 'bad', 'awful', 'worst' ) to my $blacklist = qr/bad|awful|worst/; and then replacing the inner loop with if ( $keywords[$i] =~ $blacklist ) ....

answered May 24, 2013 at 9:39

AdrianHHH

14.1k16 gold badges55 silver badges96 bronze badges

Comments

dlaehnemann · Accepted Answer · 2013-05-24 09:32:48Z

0

This should do it:

my @indices;
for my $i (0..$#keywords) {
  for my $bl (@blacklist) {
    if ($keywords[$i] =~ $bl) {
      push(@indices, $i);
      last;
    }
  }
}
for my $i (@indices) {
  @keywords = splice(@keywords, $i);
}

answered May 24, 2013 at 9:32

dlaehnemann

7216 silver badges18 bronze badges

Collectives™ on Stack Overflow

is any of array items contained in a string

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related