How do I troubleshoot and improve this slow running query? - performance

Question

I'm looking to fine tune a string search query that I am using on Mongo. In the SQL Server world, I'd like to believe I have a decent understanding of how indexes work and how to build proper indexes. I tried giving it a shot with Mongo, but, I don't believe that I'm not going about it the right way.
My collection has roughly 4.3 million documents. The document structure looks like this:
{
"_id":{
"$oid":"527027456239d1212c07a621"
},
"ReleaseId":2451,
"Status":"Accepted",
"Title":"Hard Rhythmic Motions",
"Country":"US",
"MasterId":"35976",
"Images":[
{
"Type":"primary",
"URI":"http://api.discogs.com/image/R-2451-1117047026.jpg",
"URI150":"http://api.discogs.com/image/R-150-2451-1117047026.jpg",
"Height":307,
"Width":307
},
{
"Type":"secondary",
"URI":"http://api.discogs.com/image/R-2451-1117047033.jpg",
"URI150":"http://api.discogs.com/image/R-150-2451-1117047033.jpg",
"Height":307,
"Width":307
}
],
"Artists":[
{
"_id":2894,
"Name":"DJ Hyperactive"
}
],
"Formats":[
{
"Name":null,
"Quantity":1
}
],
"Genres":[
"Electronic"
],
"Styles":[
"Hardcore",
"Acid"
]
}
I am executing a case insensitive search on one of the top-level document properties and on one of the nested document properties:
db.releases.find({$or: [{Title: new RegExp('.*mozart.*',"i")},{'Artists.Name': new RegExp('.*mozart.*',"i")}]})
I tried creating an index; when I execute .getIndexes() I can see the index I created:
{
"v" : 1,
"key" : {
"Title" : 1,
"Artists.Name" : 1
},
"ns" : "discogs.releases",
"name" : "Title_1_Artists.Name_1"
}
At this point I thought that I would be all set. However, the query ends up taking between 28 and 32 seconds to execute. I tried calling .explain() to get a little more insight:
{
"cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 4098,
"nscannedObjects" : 4292400,
"nscanned" : 4292400,
"nscannedObjectsAllPlans" : 4292400,
"nscannedAllPlans" : 4292400,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 29,
"nChunkSkips" : 0,
"millis" : 29958,
"indexBounds" : {
},
"server" : "lambic:27017"
}
From my limited knowledge of Mongo, this looks like a table scan which is why the query isn't performing very well. However, I don't know how to make this query better! I would expect that the index that I created to cover this query, but, that must not be the case.
Now, the last thing I want to point out is that this is certainly not on the most robust server. The hardware specs (including CPU and RAM) are very limited. However, if my analysis is correct and I'm doing a table scan, there must be some performance improvements I can make on the Mongo side.

Answer 1

A fulltext index is probably what you need. You could also parse the document before inserting it, and put the keywords in an array inside the document and index this array.

Answer 2

Thanks everyone for the responses. I wanted to follow up on this question since it had a few votes and to make sure anyone who stumbles upon this page in the future knows what I ended up doing.
The fulltext index sounds like a great solution. However, because this is only a small side-project of mine I'm not willing to throw more hardware at the architecture (the fulltext index requires a good amount of disk space for 4 million records).
What I ended up doing is flattening my data structures to make them easier to query and removed the wildcard search so that my indexes on that new structure can actually be used. By doing this I can achieve an indexOnly query (although the performance still isn't amazing, I find it to be adequate given my weak hardware stack).