How to match search strings to content in python

Question

Usually when we search, we have a list of stories, we provide a search string, and expect back a list of results where the given search strings matches the story.

What I am looking to do, is the opposite. Give a list of search strings, and one story and find out which search strings match to that story.

Now this could be done with re but the case here is i wanna use complex search queries as supported by solr. Full details of the query syntax here. Note: i wont use boost.

Basically i want to get some pointers for the doesitmatch function in the sample code below.

def doesitmatch(contents, searchstring):
    """
    returns result of searching contents for searchstring (True or False)
    """
    ???????
    ???????


story = "big chunk of story 200 to 1000 words long"
searchstrings = ['sajal' , 'sajal AND "is a jerk"' , 'sajal kayan' , 'sajal AND (kayan OR bangkok OR Thailand OR ( webmaster AND python))' , 'bangkok']

matches = [[searchstr] for searchstr in searchstrings if doesitmatch(story, searchstr) ]

Edit: Additionally would also be interested to know if any module exists to convert lucene query like below into regex:

sajal AND (kayan OR bangkok OR Thailand OR ( webmaster AND python) OR "is a jerk")

sajal · Accepted Answer · 2009-07-09 18:01:33Z

2

After extensive googling, i realized what i am looking to do is a Boolean search.

Found the code that makes regex boolean aware : http://code.activestate.com/recipes/252526/

Issue looks solved for now.

answered Jul 9, 2009 at 18:01

sajal

7921 gold badge7 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Lennart Regebro · Accepted Answer · 2009-07-09 13:25:06Z

0

Probably slow, but easy solution:

Make a query on the story plus each string to the search engine. If it returns anything, then it matches.

Otherwise you need to implement the search syntax yourself. If that includes things like "title:" and stuff this can be rather complex. If it's only the AND and OR from your example, then it's a recursive function that isn't too hairy.

answered Jul 9, 2009 at 13:25

Lennart Regebro

173k45 gold badges230 silver badges254 bronze badges

1 Comment

sajal Over a year ago

The problem on using my search engine(solr) for this is that the in code above the list searchstrings would have 10,000s of phrases. It would not be ideal to hit the search server 10,000s of times per story. would be very expensive. Im not using any complex stuff, only: AND, OR, Quotes and Brackets Im thinking of writing a function to convert it to regex, but given my limited regex skills i thought to investigate if such a function already exists in python...

thomas · Accepted Answer · 2009-07-09 13:42:29Z

0

Some time ago I looked for a python implementaion of lucene and I came accross of Woosh which is a pure python text-based research engine. Maybe it will statisfy your needs.

You can also try pyLucene, but i did'nt investigate this one.

answered Jul 9, 2009 at 13:42

thomas

1,9154 gold badges17 silver badges19 bronze badges

Comments

Vinay Sajip · Accepted Answer · 2009-07-09 14:39:00Z

0

Here's a suggestion in pseudocode. I'm assuming you store a story identifier with the search terms in the index, so that you can retrieve it with the search results.

def search_strings_matching(story_id_to_match, search_strings):
    result = set()
    for s in search_strings:
        result_story_ids = query_index(s) # query_index returns an id iterable
        if story_id_to_match in result_story_ids:
            result.add(s)
    return result

edited Jul 9, 2009 at 14:39

answered Jul 9, 2009 at 14:15

Vinay Sajip

100k15 gold badges184 silver badges196 bronze badges

3 Comments

sajal Over a year ago

The problem is that my index is solr running on another server, and search_strings would have over 10,000+ terms in it. running so many queries would be expensive in terms of time and resources.

Vinay Sajip Over a year ago

How often do the search strings change?

sajal Over a year ago

several times a day (not yet fully decided its upcoming project) ... but > 1ce/hour

itsadok · Accepted Answer · 2009-07-10 14:25:31Z

0

This is probably less interesting to you now, since you've already solved your problem, but what you're describing sounds like Prospective Search, which is what you call it when you have the query first and you want to match it against documents as they come along.

Lucene's MemoryIndex is a class that was designed specifically for something like this, and in your case it might be efficient enough to run many queries against a single document.

This has nothing to do with Python, though. You'd probably be better off writing something like this in java.

answered Jul 10, 2009 at 14:25

itsadok

29.5k32 gold badges133 silver badges176 bronze badges

2 Comments

sajal Over a year ago

looks interesting, i am already using solr(lucene based) for some stuff, ill see if it can be made into using this. The reason id prefer it to be in python is because im using it within a django project. moreover i cant even write hello world in java :)

Herman Bovens Over a year ago

I know this is an old question/comment, but people reading this may be interested in the fact that ElasticSearch provides this with its Percolation feature.

Bob Wyman · Accepted Answer · 2011-08-22 19:44:27Z

0

If you are writing Python on AppEngine, you can use the AppEngine Prospective Search Service to achieve exactly what you are trying to do here. See: http://code.google.com/appengine/docs/python/prospectivesearch/overview.html

answered Aug 22, 2011 at 19:44

Bob Wyman

1012 bronze badges

Collectives™ on Stack Overflow

How to match search strings to content in python

6 Answers 6

Comments

1 Comment

Comments

3 Comments

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

1 Comment

Comments

3 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related