Query multiple values at a time pymongo

Question

Currently I have a mongo document that looks like this:

{'_id': id, 'title': title, 'date': date}

What I'm trying is to search within this document by title, in the database I have like 5ks items which is not much, but my file has 1 million of titles to search.

I have ensure the title as index within the collection, but still the performance time is quite slow (about 40 seconds per 1000 titles, something obvious as I'm doing a query per title), here is my code so far:

Work repository creation:

class WorkRepository(GenericRepository, Repository):
    def __init__(self, url_root):
        super(WorkRepository, self).__init__(url_root, 'works')
        self._db[self.collection].ensure_index('title')

The entry of the program (is a REST api):

start = time.clock()
for work in json_works: #1000 titles per request
    result = work_repository.find_works_by_title(work['title'])

    if result:
        works[work['id']] = result

end = time.clock()
print end-start

return json_encoder(request, works)

and find_works_by_title code:

def find_works_by_title(self, work_title):
    works = list(self._db[self.collection].find({'title': work_title}))

    return works

I'm new to mongo and probably I've made some mistake, any recommendation?

Max Noel · Accepted Answer · 2014-11-18 13:41:07Z

20

You're making one call to the DB for each of your titles. The roundtrip is going to significantly slow the process down (the program and the DB will spend most of their time doing network communications instead of actually working).

Try the following (adapt it to your program's structure, of course):

# Build a list of the 1000 titles you're searching for.
titles = [w["title"] for w in json_works]

# Make exactly one call to the DB, asking for all of the matching documents.
return collection.find({"title": {"$in": titles}})

Further reference on how the $in operator works: http://docs.mongodb.org/manual/reference/operator/query/in/

If after that your queries are still slow, use explain on the find call's return value (more info here: http://docs.mongodb.org/manual/reference/method/cursor.explain/) and check that the query is, in fact, using an index. If it isn't, find out why.

answered Nov 18, 2014 at 13:41

Max Noel

8,9301 gold badge30 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

KBorja Over a year ago

Will try, but looks exactly what I was looking for :)

KBorja Over a year ago

When using the in operator of course the performance increases, however I'm not getting any match (I'm adding a known value in every sequence of titles to make sure something matches) but not lucky until now.

KBorja Over a year ago

Ok, my bad, used multiple [[ when generating the list, worked perfectly.

Collectives™ on Stack Overflow

Query multiple values at a time pymongo

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related