Very strange -- Adding a compound index makes queries much slower (MongoDB) - performance

Question

I'm having a problem that should be very simple but I'm stumped on this one -- maybe I'm misunderstanding something about compound indexes in MongoDB.
To reproduce this problem, I have created a simple collection with 500000 entries and six fields, each with a random number. In a mongo terminal, I generated the collection like this:
for(i = 0; i < 500000; i++){
db.test.save({a: Math.random(), b: Math.random(), c: Math.random(), d: Math.random(), e: Math.random() })
}
Then, I time a simple query on this collection like this:
t1 = new Date().getTime()
db.test.count({a : {$gt: 0.5}, b : {$gt: 0.5}, c : {$gt: 0.5}, d : {$gt: 0.5}, e : {$gt: 0.5} })
t2 = new Date().getTime()
t2-t1
=> 335ms
The query completed in 335 ms. So now I add a compound index to try to make the query faster:
db.test.ensureIndex({a: 1, b:1 ,c:1 ,d:1, e:1})
The query should be faster now, but running the exact same query takes longer:
t1 = new Date().getTime()
db.test.count({a : {$gt: 0.5}, b : {$gt: 0.5}, c : {$gt: 0.5}, d : {$gt: 0.5}, e : {$gt: 0.5} })
t2 = new Date().getTime()
t2-t1
=> 762ms
The same query takes over twice as long when the index is added! This is repeatable even when I try this multiple times. Removing the index with db.test.dropIndexes() makes the query run faster again, back to ~350ms.
Checking the queries with explain() shows that a BasicCursor is used before the index is added. After the index is added a BtreeCursor is used and has the expected indexBounds.
So my question is: why is this happening? And more importantly, how DO I get this query to run faster? In a SQL benchmark that I did on the same machine, an analogous query with SQL took ~240ms without an index, with an index dropping that down to ~180ms.
My MongoDB version info:
> mongo --version
MongoDB shell version: 2.6.3

Answer 1

The problem with your example here is basically that the data in indeed far "too random" in order to make effective use of an index in this case. The result is as expected since there is not much "order" in how an index can traverse this, along with the consideration that as you are indexing every field in the document the index size will be somewhat larger than the document itself.
For a better representation of a "real world" situation you can look at a more 50/50 split of the relevant data to search for. Here with a more optimized form of generator:
var samples = [{ "a": "a", "b": "a" },{ "a": "b", "b": "b" }];
for ( var x = 0; x < 5; x++ ) {
samples.forEach(function(s) {
var batch = [];
for(i = 0; i < 10000; i++){
batch.push( s );
}
db.test.insert(batch);
});
}
That inserts the data with a fair enough representation that either search would essentially have to scan through every document in the collection in certainty to retrieve them all in absence of an index.
So if you look a a query now with a form to get 50% of the data:
db.test.find({ "a": 1, "b": 1 }).explain()
On my hardware where I am sitting, even warmed up that is going to consistently take over 100ms to complete. But when you add an index to both fields:
db.test.ensureIndex({ "a": 1, "b": 1 })
Then the same query consistently completes under 100ms, and mostly around the 90ms mark. This also gets a little more interesting when you add some projection in order to force the stats to "index only":
db.test.find({ "a": 1, "b": 1 },{ "_id", "a": 1, "b": 1 }).explain()
Now while this does not need to go back to the documents in this case and is marked as "indexOnly": true, the working set size is likely small enough to fit in memory and thus you see a slight performance degradation due to the extra work "projecting" the fields. The average now with the index is around 110ms on the hardware. But when you drop the index:
db.test.dropIndexes()
The performance of the query without the use of an index drops to 170ms. This shows the overhead in projection against the benefits of the index more clearly.
Pulling the index back to the form as you had originally:
db.test.ensureIndex({ "a": 1, "b": 1, "c": 1, "d": 1, "e": 1 })
Keeping the same projection query you get around 135ms with the index and of course the same 170ms without. Now if you then go back to the original query form:
db.test.find({ "a": 1, "b": 1, "c": 1, "d":1, "e": 1}).explain()
The results with the index are still around the 135ms mark and the non-indexed query is skipping around the 185ms mark.
So it does make sense that real world data distribution is not typically so "random" as the test you designed. Though it is also true that distribution is almost never as clear cut as 50/50, the general case is there is not in fact so much scatter and there tends to be natural clusters of the ranges you are looking for.
This also serves as an example that with "truly random" data with a high level of distribution between values, then b-tree indexes are not the most optimal way to address the accessing of data.
I hope that makes some of the points to consider about this more clear to you.
Here is another sample closer to your original test, the only difference is altering the "precision" so the data is not so "random" which was one of the main points I am making:
var batch = []
for( i = 0; i < 500000; i++){
batch.push({
"a": Math.round(Math.random()*100)/100,
"b": Math.round(Math.random()*100)/100,
"c": Math.round(Math.random()*100)/100,
"d": Math.round(Math.random()*100)/100,
"e": Math.round(Math.random()*100)/100
});
if ( batch.length % 10000 == 0 ) {
db.test.insert( batch );
batch = [];
}
}
So there is a "two decimal place precision" in the data being enforced which again represents real world data cases more directly. Also note that the inserts are not being done on every iteration, as the implementation of insert for the shell in MongoDB 2.6 will return the "write concern" response with every update. So much faster to set up.
If you then consider your original test query, the response without an index will take around 590ms to complete as per my hardware. When you add the same index the query completes in 360ms.
If you do that on just "a" and "b" without an index:
db.test.find({ "a": {"$gt": 0.5}, "b": {"$gt": 0.5} }).explain()
The response comes in at around 490ms. Adding an index to just "a" and "b"
db.test.ensureIndex({ "a": 1, "b": 1 })
And the indexed query takes around 300ms, so still considerably faster.
Everything here says essentially:
Natural distribution is supported very well with B-tree indexes, fully random is not.
Index what you need to query on an those fields only. There is a size cost and there is a memory cost as well.
From that second point there is one more thing to demonstrate as most examples here are generally required to look up the document from the collection as well as find it in the index. The obvious cost here is that both the index and the collection need to be paged into memory in order to return the results. This of course takes time.
Consider the full compound index in place with the following query, the response without the index takes around 485ms:
db.test.find({ "a": {"$gt": 0.5}, "b": {"$gt": 0.5} }).explain()
Adding the compound index on "a" through "e" makes the same query with the index in place run around 385ms.Still faster, but slower than our full query, but there is a good reason why considering the index contains all of the fields and the conditions. But if you alter that with a projection for only the required fields:
db.test.find(
{ "a": {"$gt": 0.5}, "b": {"$gt": 0.5} },
{ "_id": 0, "a": 1, "b": 1 }
).explain()
That drops the time somewhat and now the index is used soley to get the results. Dropping the index and issuing the same query takes around 650ms with the additional overhead of the projection. This shows that an effective index actually does make a lot of difference to the results.