MongoDB geospatial query with sort - performance issues - performance

Question

I have query (which is very slow ~2,5s):
db.markers.find({ latlng: { '$within': { '$box': [ [ -16, -140 ], [ 75, 140 ] ] } } }).sort({_id: -1}).limit(1000)
When I run explain for this query I get
{
"cursor" : "GeoBrowse-box",
"isMultiKey" : false,
"n" : 1000,
"nscannedObjects" : 242331,
"nscanned" : 242331,
"nscannedObjectsAllPlans" : 242331,
"nscannedAllPlans" : 242331,
"scanAndOrder" : true,
"indexOnly" : false,
"nYields" : 1383,
"nChunkSkips" : 0,
"millis" : 2351,
"indexBounds" : {
"latlng" : [ ]
},
"lookedAt" : NumberLong(262221),
"matchesPerfd" : NumberLong(242331),
"objectsLoaded" : NumberLong(242331),
"pointsLoaded" : NumberLong(0),
"pointsSavedForYield" : NumberLong(0),
"pointsChangedOnYield" : NumberLong(0),
"pointsRemovedOnYield" : NumberLong(0),
"server" : "xx:27017"
}
When I remove sort({_id: -1}) explain gives me (fast query 5 milis):
{
"cursor" : "GeoBrowse-box",
"isMultiKey" : false,
"n" : 1000,
"nscannedObjects" : 1000,
"nscanned" : 1000,
"nscannedObjectsAllPlans" : 1000,
"nscannedAllPlans" : 1000,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 5,
"indexBounds" : {
"latlng" : [ ]
},
"lookedAt" : NumberLong(1000),
"matchesPerfd" : NumberLong(1000),
"objectsLoaded" : NumberLong(1000),
"pointsLoaded" : NumberLong(0),
"pointsSavedForYield" : NumberLong(0),
"pointsChangedOnYield" : NumberLong(0),
"pointsRemovedOnYield" : NumberLong(0),
"server" : "xx:27017"
}
I have 2d index on latlng, desc index on _id and compound indexes.
db.markers.ensureIndex({latlng: '2d', _id:-1})
db.markers.ensureIndex({ latlng: '2d' })
db.markers.ensureIndex({ _id: -1 })
What I want to achieve is to get markers from a particular area sorted from newest.
Any ideas or suggestions how to do a lot less than 2.5 seconds??
If someone wants to do their own tests
var i = 0,
lat = 0,
lng = 0;
for (i; i < 260000; i++) {
lat = parseFloat(Math.min(-90 + (Math.random() * 180), 90).toFixed(6));
lng = parseFloat(Math.min(-180 + (Math.random() * 360), 180).toFixed(6));
collection.insert({latlng: [lat, lng]}, function () {});
}
collection.find({ latlng: { '$within': { '$box': [ [ -90, -180 ], [ 90, 180 ] ] } } }, {latlng: 1, _id: 1 }).sort({_id: -1}).limit(1000).explain()
On my local machine I receives (~ 2,6s):
{
"cursor" : "GeoBrowse-box",
"isMultiKey" : false,
"n" : 1000,
"nscannedObjects" : 260000,
"nscanned" : 260000,
"nscannedObjectsAllPlans" : 260000,
"nscannedAllPlans" : 260000,
"scanAndOrder" : true,
"indexOnly" : false,
"nYields" : 1612,
"nChunkSkips" : 0,
"millis" : 2613,
"indexBounds" : {
"latlng" : [ ]
},
"lookedAt" : NumberLong(260000),
"matchesPerfd" : NumberLong(260000),
"objectsLoaded" : NumberLong(260000),
"pointsLoaded" : NumberLong(0),
"pointsSavedForYield" : NumberLong(0),
"pointsChangedOnYield" : NumberLong(0),
"pointsRemovedOnYield" : NumberLong(0),
"server" : "xx:27017"
}
Thx

Answer 1

Do you actually have the following three indexes defined on your collection?
db.markers.ensureIndex({ latlng: '2d', _id:-1 })
db.markers.ensureIndex({ latlng: '2d' })
db.markers.ensureIndex({ _id: -1 })
The geospatial indexing docs advise against creating multiple geo indexes on the same collection. Although MongoDB will allow it, the behavior may be undesirable. My guess for your case is that the non-compound {latlng: '2d'} may have been selected for use instead of the compound index. The explain() output doesn't really help us here, since it simply reports GeoBrowse-box instead of the index name; however, I would suggest manually hinting that the cursor use the compound index and see if the results improve. Alternatively, simply get rid of the non-compound index, so {latlng: '2d', _id:-1} because the obvious and only choice for the query optimizer.
Lastly, the {_id: -1} index is redundant and can be removed. Per the compound index documentation, direction is only relevant when dealing with indexes comprised of multiple fields. For a single-key index, we can walk the index backwards or forwards easily enough. Since MongoDB already creates an {_id: 1} index for us by default, it's more efficient to simply rely on that.
Now, with indexing out of the way: one caveat with your query is that limits are applied to the geospatial query component before sorting by non-geo criteria (_id in your case). I believe this means that, while your results will indeed be sorted by _id, that sort may not be considering all documents within the matched bounds. This is mentioned in the compound index bit of the documentation, which references SERVER-4247 as a pending solution.
Edit: Following up with your benchmark
I populated the example data, which are 260k random points between ±90 and ±180. I then ran your query:
db.markers.find(
{ latlng: { $within: { $box: [[-90, -180], [90, 180]] }}},
{ latlng: 1, _id: 1 }
).sort({_id: -1}).limit(1000).explain()
That took 1713ms (I'll use that as a baseline of comparison instead of your time of 2351ms). I'll also note that the query matched all 260k documents, and scanned the same number of index entries. It appears the limit didn't factor in until the _id sort, which is not what I would have expected based on the note here. I then tweaked the query a bit to examine some other cases:
Original query without the _id sort and limit: nscanned is 260k and time is 1470ms.
Original query without the _id sort: nscanned is 1000 and time is 9ms.
Original query without the limit: nscanned is 260k and time is 2567ms.
I also wanted to test sorting on an unindexed field alone to simulate what might happen for the _id sort after a geo match; however, I couldn't use _id since the default index will always exist. To do this, I deleted the compound geo index and then sorted by the latlng object. This resulted in nscanned of 260k and a time of 1039ms. If I add a limit of 1000, the time was 461ms.
If we add that to the 1470ms above (geo query without a sort and limit), it's very close to the original query without a limit, which was 2567ms. Likewise, if we add 461ms (limited sort) to 1470ms, it's near the original benchmark result of 1713ms. Based on that correlation, I'd wager that the _id sort in your benchmark isn't taking advantage of the compound index at all.
In any event, one other reason the benchmark is slow is due to a very wide geo match. Tighter bounds would definitely result in less data to sort, even with that sort being unindexed. That said, I do think SERVER-4247 would help you, since it would likely process the non-geo sort first before performing the geo match.

Answer 2

Are your indexes using compound keys?
db.markers.ensureIndex({latlng: '2d', _id:-1})