Filtering textual data in Python

Question

I'm having a problem understanding what I'm doing wrong here. I have the code below (fairly straightforward).

def compileWordList(textList, wordDict):
    '''Function to extract words from text lines exc. stops,
        and add associated line nums'''
    i = 0;
    for row in textList:
        i = i + 1
        words = re.split('\W+', row)
        for wordPart in words:
            word = repr(wordPart)
            word = word.lower()
            if not any(word in s for s in stopsList):
                if word not in wordDict:
                    x = wordLineNumsContainer()
                    x.addLineNum(i)
                    wordDict[word] = x
                elif word in wordDict:
                    lineNumValues = wordDict[word]
                    lineNumValues.addLineNum(i)
                    wordDict[word] = lineNumValues
            elif any(word in s for s in stopsList):
                print(word)

The code takes a string (sentence) from a list. It then splits the string for whole words using a re.split() method, returning a list of strings (the words).

I then force the string to lower case. Then I want it to check the word for existence in a list of stop-words I have (words too common in English to bother with). The part that checks if the word is in stopsList never seems to work, as stops words end up in my wordDict every time. Also I added the bottom print(word) statement in order to check it was catching them, but nothing ever gets printed :(

There are hundreds of stop words being used in the strings passing through.

Please can someone enlighten me here? Why are the strings never getting filtered for being stop words?

Many thanks, Alex

Immediate red flags: any(word in s for s in stopsList) and any(word in s for s in stopsList). Read them again. These expressions make little sense! — Santa
– Santa, Commented Jun 30, 2011 at 0:18
What's the purpose of "word = repr(wordPart)"? I will convert "Hello" to "'Hello'", i.e. a string which contain quotes, so the words will never match your stopwords unless the latter also entered as "'a'", "'the'" etc. Apart from that, there are obvious problems with "any(word in s for s in stopsList)", as was pointed out by Santa. I think it should be just "if word not in stopsList:" — Sergey
– Sergey, Commented Jun 30, 2011 at 0:43
@Sergey I had if word not in list before but it wasn't working either so changed it. Is still valid, although the issues you talk about make sense, thanks. — Alex
– Alex, Commented Jun 30, 2011 at 1:13
@AlexW I guess "if word not in list" didn't work because of the repr(). Also, the logic of "any(word in s for s in stopsList)" is a bit different and probably not what you want - it checks if word is a substring of any of the words in the stopsList - i.e. if your stopsList is ["apple", "banana"], it would exclude words "a", "p", "l", "e", "app", "nana" etc. But, as I said, because of the repr() all your word were quoted so they didn't match anything. — Sergey
– Sergey, Commented Jun 30, 2011 at 2:44

Johannes Weiss · Accepted Answer · 2011-06-30 00:46:51Z

7

What about that?

from collections import defaultdict
import re

stop_words = set(['a', 'is', 'and', 'the', 'i'])
text = [ 'This is the first line in my text'
       , 'and this one is the second line in my text'
       , 'I like texts with three lines, so I added that one'
       ]   
word_line_dict = defaultdict(list)

for line_no, line in enumerate(text, 1): 
    words = set(map(str.lower, re.split('\W+', line)))
    words_ok = words.difference(stop_words)
    for wok in words_ok:
        word_line_dict[wok].append(line_no)

print word_line_dict

many thanks to Gnibbler: better way of writing the for-loop & more pythonic way of handling first insertion into the dict.

which prints (except the formatting of the dict)

{ 'added': [3]
, 'like': [3]
, 'that': [3]
, 'this': [1, 2]
, 'text': [1, 2]
, 'lines': [3]
, 'three': [3]
, 'one': [2, 3]
, 'texts': [3]
, 'second': [2]
, 'so': [3]
, 'in': [1, 2]
, 'line': [1, 2]
, 'my': [1, 2]
, 'with': [3]
, 'first': [1]
}

edited Jun 30, 2011 at 0:46

answered Jun 30, 2011 at 0:30

Johannes Weiss

54.4k16 gold badges105 silver badges139 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Alex Over a year ago

Wow that is really cool thank you. I will check have a go tomorrow. Thanks again!

John La Rooy Over a year ago

In Python2.6+ you can use for line_no, line in enumerate(text, 1):

John La Rooy Over a year ago

Also has_key is deprecated (python3 dicts don't have it). You should just use if not wok in word_line_dict:. Better yet make word_line_dict a defaultdict(list)

Swiss Over a year ago

If you are using Python 2.7 you can replace the map with a set comprehension, which is a bit more pythonic. words = {word.lower() for word in re.split('\W+', line)}

Collectives™ on Stack Overflow

Filtering textual data in Python

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related