MongoDB performance of db.eval() in javascript shell vs. pymongo

Question

I need to perform update operations on documents in which I need to rotate some values in an array. MongoDB update queries currently do not allow you to $pop and then $push on the same field in an update. After searching for advice online, I decided that db.eval() would be most appropriate for my use since it ensures atomicity and the operation that I'm performing is very short so it wont lock up the db for too long.

Here's an example of something I'm trying to do:

db.eval(function (id, newVal) {
    doc = db.collection.findOne({_id: id});
    doc.values.shift();
    doc.values.push(newVal);
    db.collection.save(doc);
}, id, newVal);

And this works perfectly! I then enabled mongoDB profiling to see how many milliseconds the eval() command was taking and I'd always get results of less that 1 millisecond:

 > db.system.profile.find({op: "command"}, {"millis": 1}) 
 { "millis" : 0 }
 { "millis" : 0 }
 ...

This is good news to me, except that my application is in python, so I'm using a pymongo client to perform the eval() commands. (The data above is from the mongo shell) But now, when I run identical eval() commands using pymongo:

conn = pymongo.Connection(mongo_server_hostname)
db = conn.my_db

db.eval("""function (id, newVal) {
    doc = db.collection.findOne({_id: id});
    doc.values.shift();
    doc.values.push(newVal);
    db.collection.save(doc);
}""", id, new_val)

I get very different profiling results:

> db.system.profile.find({op: "command"}, {"millis": 1}) 
{ "millis" : 13 }
{ "millis" : 14 }
{ "millis" : 14 }
...

Is there something fundamentally different about running the same eval() commands from within the mongo shell and pymongo that results in the server taking 14ms more to run identical commands from pymongo?

Can you show your python code that is equivalent? How are you calling the eval from pymongo? Is it taking the same type of eval locks the shell takes? — WiredPrairie
– WiredPrairie, Commented Mar 15, 2013 at 0:08
Edited the question. I assume pymongo's eval() results in the same eval command. Nothing I've come across has led me to believe otherwise. — jlhawn
– jlhawn, Commented Mar 15, 2013 at 0:29
What is your MongoDB and PyMongo versions? The answer can be in the writeconcern settings... — Victor Schröder
– Victor Schröder, Commented Mar 15, 2013 at 21:31
BTW I was analysing the Database Class from PyMongo. db.eval calls db.command which has an argument check=True as default. According to the file: 'check' (optional): check the response for errors, raising :class:'~pymongo.errors.OperationFailure' if there are any. Maybe the delay is being caused by the checking procedure. — Victor Schröder
– Victor Schröder, Commented Mar 15, 2013 at 21:40
version 2.4 added support for Capped Arrays - would that solve your problem without using db.eval? docs.mongodb.org/manual/reference/operator/slice/#op._S_slice — Asya Kamsky
– Asya Kamsky, Commented Apr 15, 2013 at 3:25

rbrito · Accepted Answer · 2013-04-22 08:39:44Z

One possible cause (but not necessarily the cause) for the discrepancy that you are seeing is that both the mongo shell and the mongod server use Google's v8 Javascript engine by default (though it can be configured to use Spidermonkey as an alternative) for interpreting the commands that you give.

Google's v8 sees hotspots in Javascript code and is likely to JIT code that is frequently used.

On the other hand, vanilla PyMongo is written in pure Python, which means that it will always be interpreted, which is something that has a considerable overhead.

If you are not doing so already, one possibility would be to use the PyMongo extension written in C instead of default one or, if the rest of your application is compatible, use the PyPy JIT interpreter for Python.

If you use any distribution derived from Debian (like Ubuntu), the package python-pymongo-ext provides you the precompiled version of the C version of PyMongo.

Collectives™ on Stack Overflow

MongoDB performance of db.eval() in javascript shell vs. pymongo

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related