mongodb update causes extremly slow reads - performance

Question

I am relatively new to mongoDB.
I set up a shard mongo cluster with 2 Replica Sets; each set in a shard. -> 4 mongo deamons
The deamons are distributed on 2 WIN server with 8gb ram each.
I have a Test Collection with 10 mio documents (~ 600bytes / doc) and using the c# driver to connect to the mongos (primaryPreferred)
Now if i run some thousands single read-queries on the shard key I can see that mongo eats up more and more memory and stalls at around 7,2GB. Almost no more page faults and the queries are extremly fast. Good!
The same with more complex queries on different document properties
(Combined Index for those queries exists)
BUT
If I execute just a couple of update queries, I got a huge drop in memory usage... like mongo frees up 3GB of RAM just in no time and the so fast read queries are getting very slow.
It gets worse if i launch like 500k upserts (Save) in a row.
A complex query that was taking like 2sec to run takes now 22minutes.
I get the same behavior for Count-Queries with the same query parameters.
Is that a rather normal mongoDB behaviour or is there something that I missed to set up?
--- UPDATE #hwatkins
MongoDB version: 2.2.2
1 document scanned for a single upsert
I Have quite high disk activity during the bulk-upsert
explain() for a complex count- query before upsert
Count Explain: { "clusteredType" : "ParallelSort", "shards" : { "set1/xxxx:1234,yyyy:1234" : [{ "cursor" : "BtreeCursor AC", "isMultiKey" : false, "n" : 20799, "nscannedObjects" : 292741, "nscanned" : 292741, "nscannedObjectsAllPlans" : 294290, "nscannedAllPlans" : 294290, "scanAndOrder" : false, "indexOnly" : false, "nYields" : 2, "nChunkSkips" : 0, "millis" : 2382, "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] }, "allPlans" : [{ "cursor" : "BtreeCursor AC", "n" : 20795, "nscannedObjects" : 292741, "nscanned" : 292741, "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] } }, { "cursor" : "BasicCursor", "n" : 4, "nscannedObjects" : 1549, "nscanned" : 1549, "indexBounds" : { } }], "oldPlan" : { "cursor" : "BtreeCursor AC", "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] } }, "server" : "xxxx:1234" }], "set2/xxxx:56789,yyyy:56789" : [{ "cursor" : "BtreeCursor AC", "isMultiKey" : false, "n" : 7000, "nscannedObjects" : 97692, "nscanned" : 97692, "nscannedObjectsAllPlans" : 98941, "nscannedAllPlans" : 98941, "scanAndOrder" : false, "indexOnly" : false, "nYields" : 0, "nChunkSkips" : 0, "millis" : 729, "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] }, "allPlans" : [{ "cursor" : "BtreeCursor AC", "n" : 6996, "nscannedObjects" : 97692, "nscanned" : 97692, "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] } }, { "cursor" : "BasicCursor", "n" : 4, "nscannedObjects" : 1249, "nscanned" : 1249, "indexBounds" : { } }], "oldPlan" : { "cursor" : "BtreeCursor AC", "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] } }, "server" : "yyyy:56789" }] }, "cursor" : "BtreeCursor AC", "n" : 27799, "nChunkSkips" : 0, "nYields" : 2, "nscanned" : 390433, "nscannedAllPlans" : 393231, "nscannedObjects" : 390433, "nscannedObjectsAllPlans" : 393231, "millisShardTotal" : 3111, "millisShardAvg" : 1555, "numQueries" : 2, "numShards" : 2, "millis" : 2384 }
explain() after upsert for the same query
{ "clusteredType" : "ParallelSort", "shards" : { "set1/xxxx:1234,yyyy:1234" : [{ "cursor" : "BtreeCursor AC", "isMultiKey" : false, "n" : 20799, "nscannedObjects" : 292741, "nscanned" : 292741, "nscannedObjectsAllPlans" : 294290, "nscannedAllPlans" : 294290, "scanAndOrder" : false, "indexOnly" : false, "nYields" : 379, "nChunkSkips" : 0, "millis" : 391470, "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] }, "allPlans" : [{ "cursor" : "BtreeCursor AC", "n" : 20795, "nscannedObjects" : 292741, "nscanned" : 292741, "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] } }, { "cursor" : "BasicCursor", "n" : 4, "nscannedObjects" : 1549, "nscanned" : 1549, "indexBounds" : { } }], "server" : "xxxx:1234" }], "set2/xxxx:56789,yyyy:56789" : [{ "cursor" : "BtreeCursor AC", "isMultiKey" : false, "n" : 7000, "nscannedObjects" : 97692, "nscanned" : 97692, "nscannedObjectsAllPlans" : 98941, "nscannedAllPlans" : 98941, "scanAndOrder" : false, "indexOnly" : false, "nYields" : 0, "nChunkSkips" : 0, "millis" : 910, "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] }, "allPlans" : [{ "cursor" : "BtreeCursor AC", "n" : 6996, "nscannedObjects" : 97692, "nscanned" : 97692, "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] } }, { "cursor" : "BasicCursor", "n" : 4, "nscannedObjects" : 1249, "nscanned" : 1249, "indexBounds" : { } }], "oldPlan" : { "cursor" : "BtreeCursor AC", "indexBounds" : { "f.14.b" : [["A", "A"]], "f.500.b" : [[10, 50]] } }, "server" : "yyyy:56789" }] }, "cursor" : "BtreeCursor AC", "n" : 27799, "nChunkSkips" : 0, "nYields" : 379, "nscanned" : 390433, "nscannedAllPlans" : 393231, "nscannedObjects" : 390433, "nscannedObjectsAllPlans" : 393231, "millisShardTotal" : 392380, "millisShardAvg" : 196190, "numQueries" : 2, "numShards" : 2, "millis" : 391486 }
Btw:
*One single upsert (one affected doc) lets the memory usage drop by around 600MB. --> ~ 4,5GB mem usage reached only after some queries.
if i take the query from above and i use the mongoCursor to loop on the result-set it just takes ages... (query still running as i type) :(
UPDATE II #Daniel
Here you got a sample doc stored in the mongoDB-Cluster.
The Shard Key is the b -Property of my doc (it corresponds to a telephone number)
Upsert:
I search back existing docs by the shard-key and update some properties of the f- array. Then I call Save on the mongoDB driver for all those docs one by one (like 500k times).
There is an index: { "f.14.b" : 1, "f.500.b" : 1 }
This index is used for complex queries. Like described above those queries are fast before the bulk-update and extremely slow after the update.
{
"_id" : ObjectId("51248d6xxxxxxxxxxxxx"),
"b" : "33600000000",
"f" : {
"500" : {
"a" : ISODate("2013-02-20T08:45:38.075Z"),
"b" : 91
},
"14" : {
"a" : ISODate("2013-02-20T08:45:38.075Z"),
"b" : "A"
},
"1501" : {
"a" : ISODate("2013-02-20T08:45:38.141Z"),
"b" : ["X", "Y", "Z"]
},
"2000" : {
"a" : ISODate("2013-02-20T08:45:38.141Z"),
"b" : false
}
}
}
Thanks a lot,
Blume

Answer 1

This is interesting. It looks like first, your data is not very evenly distributed. Your explain shows nscanned: 292741 on the first set and nscanned: 97692 on the second set. Pretty big difference. It also shows on the first set nyields:379 and on the second set nyields:0. This implies that only are you reading unevenly from your sets, you are probably writing unevenly to them. You will get more out of your cluster if you choose a shard key that has a more even distribution.
As to why specifically this is happening with your upserts, are you adding more data to your existing documents? If so you are probably a victim of document movement. In your mongodb logs do you see any queries with moved: 1? This means the slow query in the log also had a document movement on disk which causes lots of havok with indexes into arrays/subdocuments. Mongodb I believe still will essentially have to do an index rebuild on the entire document if it moves and will have to do some heavy updating of all indexes into subdocuments/arrays.
The workaround for document movement is to preallocate extra data on document at creation then immediately remove it from the document. Mongo allocates all documents with a fixed space + padding factor on the disk. If they outgrow their space, they must be moved on the disk to a larger area. If you created your documents with already extra data then remove it, you will give yourself a lot of extra padding on disk to accomodate your document growth. This can be a waste of space for sure but it will be a big saver of performance.

Answer 2

What version of mongodb are you on?
When you do the upsert can you do an .explain() on it to see how
many documents it's scanning.
What does the disk io look like during the upserts