I have cPanel/WHM installed on a 40gb partition, however WHM shows that 8.9gb out of 9.9gb is in use. How do I correct this?
This is on an AWS EC2 instance. The root volume is configured to 40gb.
After running df -h :
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
9.9G 8.9G 574M 95% /
/dev/hda1 99M 26M 69M 28% /boot
tmpfs 1006M 0 1006M 0% /dev/shm
So that shows that the /dev/mapper/VolGroup00-LogVol00 is 9.9GB. However, if I run parted and print the configuration I can see that:
Model: QEMU HARDDISK (ide)
Disk /dev/hda: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 107MB 107MB primary ext3 boot
2 107MB 21.5GB 21.4GB primary lvm
I need the whole 40GB for cPanel/WHM. Why would it limit its self to 1/4 of the disk?
After Running vgs
VG #PV #LV #SN Attr VSize VFree
VolGroup00 1 2 0 wz--n- 19.88G 0
pvs:
PV VG Fmt Attr PSize PFree
/dev/hda2 VolGroup00 lvm2 a-- 19.88G 0
lvs:
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
LogVol00 VolGroup00 -wi-ao 10.22G
LogVol01 VolGroup00 -wi-ao 9.66G
fdisk -l
Disk /dev/hda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 * 1 13 104391 83 Linux
/dev/hda2 14 2610 20860402+ 8e Linux LVM
Disk /dev/dm-0: 10.9 GB, 10972299264 bytes
255 heads, 63 sectors/track, 1333 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1: 10.3 GB, 10368319488 bytes
255 heads, 63 sectors/track, 1260 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-1 doesn't contain a valid partition table
Where are you checking disk usages in WHM ? Can you please let me know following command out put so that I can assist you on this.
df -h
I think there is free space on your server in LVM partition. Can you please check this with the following command and let me know
vgs
pvs
lvs
fdisk -l
And if you found any free space in your VolGroup, Then you will have to increase it through lvextend command, You can check it at http://www.24x7servermanagement.com/blog/how-to-increase-the-size-of-the-logical-volume/
Related
i have a strange issue in my elasticsearch cluster
so i have 5 nodes ( 4 data and masters and 1 master only node )
so each node has 5.7 tb disk space on it
but on the first node my disk is almost completely full, and on the rest it is half full
the number of shards on all nodes is approximately the same
df -h from first node
/dev/mapper/vg1-data 5.8T 5.1T 717G 88% /var/lib/elasticsearch
and here is /cat/shards output
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
354 5tb 5tb 714.9gb 5.7tb 87 10.0.5.21 10.0.5.21 elastic-01
392 3.2tb 3.2tb 2.5tb 5.7tb 55 10.0.5.23 10.0.5.23 elastic-02
393 3.8tb 3.8tb 1.8tb 5.7tb 67 10.0.5.28 10.0.5.28 elastic-07
392 3.9tb 3.9tb 1.7tb 5.7tb 69 10.0.5.27 10.0.5.27 elastic-06
i tried summing the results from cat/shards | grep elastic-01 and it turned out that all shards on this node occupy 3.5 tb
curl -X GET http://10.0.5.22:9200/_cat/shards | grep elastic-01 | awk '{ print $6 }'
94.3kb
279b
1.4gb
333.9mb
13.2gb
260.4mb
11.5gb
20.3gb
28.5gb
10gb
12.8gb
365.7mb
9.1gb
263.3mb
92.5gb
951.1kb
266.4mb
35.9gb
10.8gb
299.6mb
22gb
526kb
31.2mb
110.1mb
1mb
46.9gb
19.3gb
358.1kb
17.9gb
22.4kb
11.7gb
3.9gb
5.1gb
427.2mb
1.1mb
48.4gb
elastic-01
75.3mb
6.7gb
30.6gb
43.8gb
31.1mb
21.3gb
10.7gb
1.1gb
17gb
5.1gb
38.4gb
49.1gb
20.2mb
7gb
7.3mb
7.3mb
383.1mb
322.7mb
130.9gb
18.5gb
34.1gb
291.8mb
537.3mb
1.6gb
15.6gb
96.4mb
7.4mb
5.8gb
114.3gb
4.3gb
25gb
7.4gb
7.4gb
638.1kb
10.5gb
175.6kb
275.9mb
33.2mb
806.8kb
35.5gb
40.1gb
17.1gb
408.6mb
115.2mb
69mb
20.3gb
542.4kb
28.4gb
385.6mb
12.9gb
1.3mb
5.5mb
66.6mb
17.5gb
18.7gb
35.6gb
10.9gb
986.3kb
10.3gb
19.1gb
412.8mb
34.4gb
22.6gb
5.1gb
883.4kb
5.3gb
10.4gb
276.4mb
31.9gb
34.5gb
58.1gb
22.3gb
18.8gb
93.9kb
176.5gb
249.3mb
38.1kb
12.1gb
19.7gb
7.6gb
24.7gb
779.9kb
11.2gb
4.9mb
19.1gb
1.2gb
21.1gb
30.4gb
3.8gb
276.5kb
26.3gb
379.9mb
10.4gb
5.5gb
31gb
802.4kb
868.3kb
43.9gb
5.8gb
463.5mb
18.7gb
3.3gb
12gb
4.3gb
32.1gb
3.3gb
11.3gb
1.2mb
944kb
118.2mb
25.8gb
23.9gb
799kb
410.4mb
6mb
5.1gb
32gb
30gb
7.8gb
32.3gb
24.9gb
25.1gb
18gb
16.4gb
1.2gb
915.2kb
4.9mb
29.2gb
59.5kb
1.3gb
150.8gb
1.6gb
11.2gb
17.4gb
439.4mb
6.3mb
21.6gb
394.9mb
26.9gb
23.5gb
43.8gb
28gb
8.9gb
19.5gb
30.3gb
31.8gb
14.7gb
19gb
34.9gb
41.3kb
63.4gb
41.8gb
22.7gb
15gb
32.6gb
281.4mb
379.5mb
8.6mb
3.6mb
37.7gb
10.9gb
818.7kb
19gb
115kb
112.3kb
10gb
7.4mb
685.2kb
332.9mb
5gb
20.2gb
39.5gb
8.6mb
289.5mb
19.3mb
289.6mb
1.1gb
1.6gb
24.8gb
18.1mb
915kb
22.4gb
5.8mb
429mb
261b
20.3gb
930.8kb
19.2gb
25.6gb
31gb
26.6gb
20.1gb
20.2gb
538.4kb
27.4gb
1.2mb
290.6mb
403.6mb
77.4mb
41.7gb
2.7gb
3gb
17.7gb
11.3gb
15.9gb
282.4mb
10.7gb
962.9kb
888.6kb
16.9gb
176.9gb
11.6gb
21.4gb
5.1mb
26.1gb
331.1mb
3.9gb
9.6gb
29.6gb
7.8gb
17.8gb
19.2gb
7.5gb
388.8mb
43.4gb
31.5gb
3gb
21.6mb
15.2gb
11.2gb
54.1gb
17.4gb
1.5gb
34.8gb
273.1mb
32.3gb
17.7gb
2.2gb
17.5gb
22.6gb
820.7kb
1gb
6.6gb
7.8mb
9.3gb
34.5gb
24.1gb
32.9gb
25.2gb
2.9gb
2.6gb
4.6mb
42.8gb
9.3gb
17.9kb
23.4gb
1.1gb
20.6gb
18.1gb
27gb
25.7gb
5mb
32.5gb
29.1gb
42kb
22.5gb
3.1mb
22.6gb
9.8gb
11gb
28.5gb
14.2gb
89.2kb
34.5gb
41.8gb
25gb
410.2mb
20.6gb
16.5gb
16.2gb
19.8gb
7.3gb
13.4gb
11.4gb
10.4gb
11.8gb
7.3mb
1.1gb
46.9gb
10.4gb
535.6mb
55.5gb
19.2gb
14.1gb
20.3gb
28.9gb
30.5gb
4.7gb
49.4gb
7.7gb
9.7gb
6.6gb
20.7gb
29.2gb
18.9gb
9.3gb
19gb
757.4kb
902.4kb
but why does both elastic and du -hs show that more space is being used?
du -hs inside /var/lib/elasticsearch shows 5.1 tb too
du-hs*
5.1T nodes
4.0K range
You should change your URL to
http://10.0.5.22:9200/_cat/shards?bytes=b
So that you get whole numbers instead of human readable ones. You probably have an issue adding up kb/mb/gb figures because when all figures are whole numbers without units, I get 5,265,110,799,379 which matches pretty well with the results you get from _cat/allocation and du -hs
As to the reason why elastic-01 uses more disk space than the other nodes, it's because it seems to have very big shards on it. From the list you shared, you can see these shard sizes (sort desc):
176 900 000 000
176 500 000 000
150 800 000 000
130 900 000 000
114 300 000 000
92 500 000 000
63 400 000 000
58 100 000 000
55 500 000 000
54 100 000 000
49 400 000 000
49 100 000 000
We can see 5 shards whose size is way above 100GB each, and this is usually not a good sign, because your shards have grown too big. Remember that shards are the unit of partitioning of your indexes.
I'm pretty sure there aren't that big shards on your other nodes.
There are a couple ways forward.
First, I would check if the index containing those shards contains time-based data and the index is not using Index Lifecycle Management.
Second, I would check if those shards contain a lot of deleted documents by looking at the docs.deleted column resulting from the following command:
GET _cat/indices?v
If that's the case (in the case the index holds documents being frequently updated), it might be possible to regain some space by running
POST <index_name>/_forcemerge?only_expunge_deletes=true
The previous command should be run with great care, because it requires disk space and you don't have much left, so it might not be possible in your case.
There are other ways, but I would first investigate these two points first.
I built a ramdisk image which is bigger than 16 MB. After downloading the ramdisk image U-boot print:
## Loading init Ramdisk from Legacy Image at 03000000 ...
Image Name: uboot ext4 ramdisk rootfs
Created: 2018-01-24 8:15:12 UTC
Image Type: AArch64 Linux RAMDisk Image (gzip compressed)
Data Size: 38319491 Bytes = 36.5 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 01000000
Booting using the fdt blob at 0x1000000
Loading Ramdisk to 3d567000, end 3f9f2583 ... OK
Using Device Tree in place at 0000000001000000, end 000000000100c554
Starting kernel ...
Timer summary in microseconds:
Mark Elapsed Stage
0 0 reset
1,106,000 1,106,000 id=64
1,437,000 331,000 id=65
1,445,000 8,000 main_loop
102,482,800,0,02,338,300,0, id=80
102,482,800,0 0 tftp_start
102,482,800,0, 0 eth_start
102,985,300,0, 5,025,000 id=81
102,985,300,0 0 id=84
102,985,300,0 0 id=82
102,985,300,0 0 tftp_done
104,389,500,0,14,042,000 bootm_start
104,734,100,0, 3,446,000 id=9
104,734,100,0 0 id=10
105,199,700,0 4,656,000 id=11
105,200,400,0 7,000 id=12
105,821,800,0 6,214,000 id=15
105,822,000,0 2,000 start_kernel
166,835,917,366,575,094,88166,835,917,355,992,894,88 board_init_f
Accumulated time:
==================================================================
Kernel is hanging.
But if the ramdisk image is smaller than 16 MB, it works.
What can I do to fix this problem?
While running memory_profiler, I noticed a large amount of memory being allocated by Ruby's net/protocol.rb component. I call it when performing an HTTP request to a server to download a file. The file is 43.67MB large and net/protocol.rb alone allocates 262,011,476 bytes just to download it.
Looking at the "allocated memory by location" section in the profiler report below, I can see net/protocol.rb:172 and http/response.rb:334 allocating 50-60MB of memory each, which is about the size of the file, so that looks reasonable. However, the top most entry (net/protocol.rb:153) worries me: that's 200MB of memory, at least 4x the size of the file.
I have two questions:
Why does net/protocol need to allocate 5x the size of the file in order to download it?
Is there anything I can do to reduce the amount of memory used by net/protocol?
memory_profiler output:
Total allocated: 314461424 bytes (82260 objects)
Total retained: 0 bytes (0 objects)
allocated memory by gem
-----------------------------------
314461304 ruby-2.1.2/lib
120 client/lib
allocated memory by file
-----------------------------------
262011476 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb
52435727 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb
7971 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb
2178 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb
1663 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb
1260 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb
949 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb
120 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb
80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/http.rb
allocated memory by location
-----------------------------------
200483909 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153
60548199 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172
52428839 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:334
978800 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:155
2537 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:61
2365 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:172
2190 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54
1280 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:56
960 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:62
836 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:165
792 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:13
738 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:125
698 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:263
576 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:214
489 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:40
480 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:127
360 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:40
328 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:610
320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:71
320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:30
320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:59
308 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:322
256 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:879
240 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1615
239 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:211
232 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:38
224 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:181
200 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:17
192 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:42
179 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:877
169 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1459
160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1029
160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:434
160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:435
160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:445
160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1617
149 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1445
147 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1529
129 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:98
128 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1475
120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:444
120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:446
120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:447
120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:29
120 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb:45
96 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:899
80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:39
80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:45
80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:46
80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:145
allocated memory by class
-----------------------------------
309678360 String
3445304 Thread::Backtrace
981096 Array
352376 IO::EAGAINWaitReadable
1960 MatchData
1024 Hash
328 Net::HTTP
256 TCPSocket
256 URI::HTTP
128 Time
120 Net::HTTP::Get
120 Net::HTTPOK
96 Net::BufferedIO
allocated objects by gem
-----------------------------------
82259 ruby-2.1.2/lib
1 client/lib
allocated objects by file
-----------------------------------
81908 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb
129 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb
127 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb
28 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb
23 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb
23 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb
19 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/http.rb
1 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb
allocated objects by location
-----------------------------------
36373 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153
24470 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:155
21057 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172
48 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:61
38 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54
32 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:56
31 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:172
24 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:62
12 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:127
9 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:40
8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:71
8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:165
8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:30
8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:59
6 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:214
6 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1615
5 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:263
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1029
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:322
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:17
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:434
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:435
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:445
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:42
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:125
4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1617
3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1529
3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:444
3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:446
3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:447
3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:40
3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1445
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:877
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:39
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:45
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:46
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:13
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:145
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:31
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:111
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:144
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:98
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:179
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:181
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:213
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1640
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1642
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:343
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:530
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:557
allocated objects by class
-----------------------------------
47935 String
24519 Array
4894 IO::EAGAINWaitReadable
4894 Thread::Backtrace
7 MatchData
3 Hash
2 URI::HTTP
1 Net::BufferedIO
1 Net::HTTP
1 Net::HTTP::Get
1 Net::HTTPOK
1 TCPSocket
1 Time
retained memory by gem
-----------------------------------
NO DATA
retained memory by file
-----------------------------------
NO DATA
retained memory by location
-----------------------------------
NO DATA
retained memory by class
-----------------------------------
NO DATA
retained objects by gem
-----------------------------------
NO DATA
retained objects by file
-----------------------------------
NO DATA
retained objects by location
-----------------------------------
NO DATA
retained objects by class
-----------------------------------
NO DATA
Allocated String Report
-----------------------------------
11926 ""
7019 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172
4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153
10 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54
2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:179
1 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:67
4894 "Resource temporarily unavailable - read would block"
4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153
4894 "UTF-8"
4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153
4894 "read would block"
4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153
...
Relevant code:
report = MemoryProfiler.report do
begin
response = nil
Net::HTTP.new(uri.host, uri.port).start { |http|
request = Net::HTTP::Get.new uri.request_uri
response = http.request request
}
rescue Net::ReadTimeout => e
raise RequestTimeoutError.new(e.message)
rescue Exception => e
raise ServerConnectionError.new(e.message)
end
end
report.pretty_print
Network traffic data from Charles proxy:
Request Header: 168 bytes
Response Header: 288 bytes
Request: -
Response: 43.67 MB (45792735 bytes)
Total: 43.67 MB (45793191 bytes)
Almost all of those strings allocated in net/protocol.rb#L153 are short-lived and are reclaimed by the next GC run. Those allocated objects are thus pretty harmless and will not result in a significantly larger process size.
You get a lot of exceptions (which are used for control flow here to read form the socket) and the actual read data which is appended to the buffer. All of these operations create temporary (internally used) objects.
As such, you are probably measuring the wrong thing. What would probably make more sense is to:
measure the maximum RSS of the process (i.e. the "used" memory);
and to measure the amount of additional memory still allocated after the read.
You will notice that (depending on the memory pressure on your computer), the RSS will not grow significantly above the amount of actually read data and that the references memory after the read is about the same size as the read data with about no internal intermediate objects still referenced.
Occasionally, we get an OutOfMemoryException in one of our IIS processes. I tried to analyze the dump but wasn't able to reach concrete conclusions. I also tried looking into MS hotfixes, found similar problems and resolutions, but not sure if its related or not: link
Below is the output of the !analyze -v command in WinDbg:
!analyze -v
[...]
CoInitialize failed 80010106
CoInitialize failed 80010106
CoInitialize failed 80010106
GetPageUrlData failed, server returned HTTP status 404
URL requested: http://watson.microsoft.com/StageOne/w3wp_exe/7_5_7601_17514/4ce7a5f8/unknown/0_0_0_0/bbbbbbb4/80000007/00000000.htm?Retriage=1
FAULTING_IP:
+75d2faf02afdbf0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000007 (Wake debugger)
ExceptionFlags: 00000000
NumberParameters: 0
BUGCHECK_STR: 80000007
PROCESS_NAME: w3wp.exe
ERROR_CODE: (NTSTATUS) 0x80000007 - {Kernel Debugger Awakened} the system debugger was awakened by an interrupt.
EXCEPTION_CODE: (HRESULT) 0x80000007 (2147483655) - Operation aborted
MOD_LIST: *** ERROR: Could not build analysis XML
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x2364 (0)
Current frame:
ChildEBP RetAddr Caller, Callee
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 370.2364 Event
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 00002364
DEFAULT_BUCKET_ID: APPLICATION_HANG_BlockedOn_EventHandle
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_BlockedOn_EventHandle
LAST_CONTROL_TRANSFER: from 758e149d to 778df8c1
FAULTING_THREAD: 00000000
STACK_TEXT:
002efb8c 758e149d 000001d4 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
002efbf8 75c71194 000001d4 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
002efc10 75c71148 000001d4 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
002efc24 7470765a 000001d4 ffffffff 747057c1 kernel32!WaitForSingleObject+0x12
002efc30 747057c1 00000000 74706f84 00a21320 w3wphost!WP_IPM::WaitForShutdown+0xb
002efc38 74706f84 00a21320 00a215d0 002efd58 w3wphost!W3WP_HOST::WaitForShutdown+0x11
002efc48 00a22bdb 002efc68 00a25708 00000001 w3wphost!AppHostInitialize+0x11e
002efd58 00a23584 0000000f 00702828 00703b48 w3wp!wmain+0x373
002efd9c 75c733aa fffde000 002efde8 778f9ed2 w3wp!_initterm_e+0x163
002efda8 778f9ed2 fffde000 71b16c75 00000000 kernel32!BaseThreadInitThunk+0xe
002efde8 778f9ea5 00a236b5 fffde000 ffffffff ntdll!__RtlUserThreadStart+0x70
002efe00 00000000 00a236b5 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
FOLLOWUP_IP:
w3wphost!WP_IPM::WaitForShutdown+b
7470765a f60520d0707403 test byte ptr [w3wphost!g_dwDebugFlags (7470d020)],3
SYMBOL_STACK_INDEX: 4
SYMBOL_NAME: w3wphost!WP_IPM::WaitForShutdown+b
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: w3wphost
IMAGE_NAME: w3wphost.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7a5d0
STACK_COMMAND: ~0s ; kb
BUCKET_ID: 80000007_w3wphost!WP_IPM::WaitForShutdown+b
FAILURE_BUCKET_ID: APPLICATION_HANG_BlockedOn_EventHandle_80000007_w3wphost.dll!WP_IPM::WaitForShutdown
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/w3wp_exe/7_5_7601_17514/4ce7a5f8/unknown/0_0_0_0/bbbbbbb4/80000007/00000000.htm?Retriage=1
Followup: MachineOwner
Additional information as requested from comments:
[0:000> !AnalyzeOOM
---------Heap 11---------
Managed OOM occured after GC #15967 (Requested to allocate 0 bytes)
Reason: Low on memory during GC
Detail: SOH: Failed to reserve memory (16777216 bytes)
---------Heap 20---------
Managed OOM occured after GC #15977 (Requested to allocate 0 bytes)
Reason: Low on memory during GC
Detail: SOH: Failed to reserve memory (16777216 bytes)
---------Heap 21---------
Managed OOM occured after GC #15979 (Requested to allocate 0 bytes)
Reason: Low on memory during GC
Detail: SOH: Failed to reserve memory (16777216 bytes)
---------Heap 22---------
Managed OOM occured after GC #15529 (Requested to allocate 0 bytes)
Reason: Low on memory during GC
Detail: SOH: Failed to reserve memory (16777216 bytes)
---------Heap 23---------
Managed OOM occured after GC #15975 (Requested to allocate 0 bytes)
Reason: Low on memory during GC
Detail: SOH: Failed to reserve memory (16777216 bytes)
---------Heap 25---------
Managed OOM occured after GC #15985 (Requested to allocate 0 bytes)
Reason: Low on memory during GC
Detail: SOH: Failed to reserve memory (16777216 bytes)
---------Heap 27---------
Managed OOM occured after GC #40008 (Requested to allocate 0 bytes)
Reason: Low on memory during GC
Detail: SOH: Failed to reserve memory (16777216 bytes)
---------Heap 30---------
Managed OOM occured after GC #40006 (Requested to allocate 0 bytes)
Reason: Low on memory during GC
Detail: SOH: Failed to reserve memory (16777216 bytes)
0:000> !vmstat
TYPE MINIMUM MAXIMUM AVERAGE BLK COUNT TOTAL
~~~~ ~~~~~~~ ~~~~~~~ ~~~~~~~ ~~~~~~~~~ ~~~~~
Free:
Small 4K 64K 57K 4,651 266,932K
Medium 68K 1,024K 288K 97 27,967K
Large 1,088K 6,080K 2,305K 27 62,247K
Summary 4K 6,080K 74K 4,775 357,150K
Reserve:
Small 4K 64K 12K 926 11,567K
Medium 68K 1,020K 277K 390 108,263K
Large 1,148K 16,376K 12,201K 190 2,318,211K
Summary 4K 16,376K 1,618K 1,506 2,438,043K
Commit:
Small 4K 64K 10K 8,169 85,567K
Medium 68K 1,024K 322K 552 178,023K
Large 1,028K 23,300K 5,137K 221 1,135,447K
Summary 4K 23,300K 156K 8,942 1,399,038K
Private:
Small 4K 64K 11K 5,939 65,578K
Medium 68K 1,024K 311K 472 146,891K
Large 1,028K 23,300K 9,725K 316 3,073,339K
Summary 4K 23,300K 488K 6,727 3,285,811K
Mapped:
Small 4K 64K 11K 85 979K
Medium 68K 1,004K 366K 12 4,399K
Large 1,520K 2,888K 2,206K 4 8,824K
Summary 4K 2,888K 140K 101 14,203K
Image:
Small 4K 64K 9K 3,071 30,575K
Medium 68K 1,024K 294K 458 134,995K
Large 1,032K 15,480K 4,082K 91 371,495K
Summary 4K 15,480K 148K 3,620 537,064K][1]
#############################
0:000> !eeheap -gc
Number of GC Heaps: 32
------------------------------
Heap 0 (1a616d08)
generation 0 starts at 0xa062179c
generation 1 starts at 0xa0621000
generation 2 starts at 0x1ab91000
ephemeral segment allocation context: none
segment begin allocated size
1ab90000 1ab91000 1adce1c8 0x23d1c8(2347464)
a0620000 a0621000 a0867db8 0x246db8(2387384)
Large object heap starts at 0x3ab91000
segment begin allocated size
3ab90000 3ab91000 3b343490 0x7b2490(8070288)
Heap Size: Size: 0xc36410 (12805136) bytes.
------------------------------
Heap 1 (1a619970)
generation 0 starts at 0xa965da00
generation 1 starts at 0xa9621000
generation 2 starts at 0x1bb91000
ephemeral segment allocation context: none
segment begin allocated size
1bb90000 1bb91000 1be9bbd0 0x30abd0(3189712)
a9620000 a9621000 a982dd14 0x20cd14(2149652)
Large object heap starts at 0x3b391000
segment begin allocated size
3b390000 3b391000 3bae09f0 0x74f9f0(7666160)
Heap Size: Size: 0xc672d4 (13005524) bytes.
------------------------------
Heap 2 (1a6215d8)
generation 0 starts at 0xa762370c
generation 1 starts at 0xa7621000
generation 2 starts at 0x1cb91000
ephemeral segment allocation context: none
segment begin allocated size
1cb90000 1cb91000 1d0a4604 0x513604(5322244)
a7620000 a7621000 a78a3a20 0x282a20(2632224)
Large object heap starts at 0x3bb91000
segment begin allocated size
3bb90000 3bb91000 3c384cf8 0x7f3cf8(8338680)
736b0000 736b1000 73769790 0xb8790(755600)
Heap Size: Size: 0x10424ac (17048748) bytes.
------------------------------
Heap 3 (1a624240)
generation 0 starts at 0xb56226d0
generation 1 starts at 0xb5621000
generation 2 starts at 0x1db91000
ephemeral segment allocation context: none
segment begin allocated size
1db90000 1db91000 1debd778 0x32c778(3327864)
b5620000 b5621000 b56346dc 0x136dc(79580)
Large object heap starts at 0x3c391000
segment begin allocated size
3c390000 3c391000 3c88b720 0x4fa720(5220128)
Heap Size: Size: 0x83a574 (8627572) bytes.
------------------------------
Heap 4 (1a626ea8)
generation 0 starts at 0x9762eb1c
generation 1 starts at 0x97621000
generation 2 starts at 0x1eb91000
ephemeral segment allocation context: none
segment begin allocated size
1eb90000 1eb91000 1ee6ae1c 0x2d9e1c(2989596)
97620000 97621000 97a87308 0x466308(4612872)
Large object heap starts at 0x3cb91000
segment begin allocated size
3cb90000 3cb91000 3d36c7b8 0x7db7b8(8239032)
f9e70000 f9e71000 f9e975a0 0x265a0(157088)
Heap Size: Size: 0xf41e7c (15998588) bytes.
------------------------------
Heap 5 (1a639b10)
generation 0 starts at 0x8f62107c
generation 1 starts at 0x8f621000
generation 2 starts at 0x1fb91000
ephemeral segment allocation context: none
segment begin allocated size
1fb90000 1fb91000 20b8500c 0xff400c(16728076)
8f620000 8f621000 8f777088 0x156088(1400968)
Large object heap starts at 0x3d391000
segment begin allocated size
3d390000 3d391000 3d903cb0 0x572cb0(5713072)
Heap Size: Size: 0x16bcd44 (23842116) bytes.
------------------------------
Heap 6 (1a63c778)
generation 0 starts at 0xba6611e8
generation 1 starts at 0xba621000
generation 2 starts at 0x20b91000
ephemeral segment allocation context: none
segment begin allocated size
20b90000 20b91000 20e66118 0x2d5118(2969880)
ba620000 ba621000 ba7051f4 0xe41f4(934388)
Large object heap starts at 0x3db91000
segment begin allocated size
3db90000 3db91000 3e348dd8 0x7b7dd8(8093144)
Heap Size: Size: 0xb710e4 (11997412) bytes.
------------------------------
Heap 7 (1a63f3e0)
generation 0 starts at 0xad621918
generation 1 starts at 0xad621000
generation 2 starts at 0x21b91000
ephemeral segment allocation context: none
segment begin allocated size
21b90000 21b91000 21fe7dd0 0x456dd0(4550096)
ad620000 ad621000 adad37e8 0x4b27e8(4925416)
Large object heap starts at 0x3e391000
segment begin allocated size
3e390000 3e391000 3eaea868 0x759868(7706728)
Heap Size: Size: 0x1062e20 (17182240) bytes.
------------------------------
Heap 8 (1a642048)
generation 0 starts at 0xf5e724e0
generation 1 starts at 0xf5e71000
generation 2 starts at 0x22b91000
ephemeral segment allocation context: none
segment begin allocated size
22b90000 22b91000 22ee2cc8 0x351cc8(3480776)
f5e70000 f5e71000 f5eb04ec 0x3f4ec(259308)
Large object heap starts at 0x3eb91000
segment begin allocated size
3eb90000 3eb91000 3f03b3c0 0x4aa3c0(4891584)
Heap Size: Size: 0x83b574 (8631668) bytes.
------------------------------
Heap 9 (1a648cb0)
generation 0 starts at 0x8d630bc4
generation 1 starts at 0x8d621000
generation 2 starts at 0x23b91000
ephemeral segment allocation context: none
segment begin allocated size
23b90000 23b91000 23e4d69c 0x2bc69c(2868892)
8d620000 8d621000 8daf7fb4 0x4d6fb4(5074868)
Large object heap starts at 0x3f391000
segment begin allocated size
3f390000 3f391000 3f991138 0x600138(6291768)
Heap Size: Size: 0xd93788 (14235528) bytes.
------------------------------
Heap 10 (1a64b918)
generation 0 starts at 0xa86261d0
generation 1 starts at 0xa8621000
generation 2 starts at 0x24b91000
ephemeral segment allocation context: none
segment begin allocated size
24b90000 24b91000 250b5b3c 0x524b3c(5393212)
a8620000 a8621000 a891ad34 0x2f9d34(3120436)
Large object heap starts at 0x3fb91000
segment begin allocated size
3fb90000 3fb91000 3ff89810 0x3f8810(4163600)
Heap Size: Size: 0xc17080 (12677248) bytes.
------------------------------
Heap 11 (1a64e580)
generation 0 starts at 0x916238ec
generation 1 starts at 0x91621000
generation 2 starts at 0x25b91000
ephemeral segment allocation context: none
segment begin allocated size
25b90000 25b91000 25ea5d64 0x314d64(3231076)
91620000 91621000 91930198 0x30f198(3207576)
Large object heap starts at 0x40391000
segment begin allocated size
40390000 40391000 40ac8f50 0x737f50(7569232)
Heap Size: Size: 0xd5be4c (14007884) bytes.
------------------------------
Heap 12 (1a65b850)
generation 0 starts at 0x7c52281c
generation 1 starts at 0x7c521000
generation 2 starts at 0x26b91000
ephemeral segment allocation context: none
segment begin allocated size
26b90000 26b91000 2702cad8 0x49bad8(4831960)
7c520000 7c521000 7c7b662c 0x29562c(2709036)
Large object heap starts at 0x40b91000
segment begin allocated size
40b90000 40b91000 41378c38 0x7e7c38(8289336)
e73d0000 e73d1000 e78cce00 0x4fbe00(5225984)
Heap Size: Size: 0x1414b3c (21056316) bytes.
------------------------------
Heap 13 (1a65ef20)
generation 0 starts at 0xf7e77370
generation 1 starts at 0xf7e71000
generation 2 starts at 0x27b91000
ephemeral segment allocation context: none
segment begin allocated size
27b90000 27b91000 27ee43d4 0x3533d4(3486676)
f7e70000 f7e71000 f828f6fc 0x41e6fc(4318972)
Large object heap starts at 0x41391000
segment begin allocated size
41390000 41391000 41b8edf0 0x7fddf0(8379888)
ebc80000 ebc81000 ec460740 0x7df740(8255296)
7e520000 7e521000 7e56dba8 0x4cba8(314280)
Heap Size: Size: 0x179bba8 (24755112) bytes.
------------------------------
Heap 14 (1a661458)
generation 0 starts at 0x9e65f268
generation 1 starts at 0x9e621000
generation 2 starts at 0x28b91000
ephemeral segment allocation context: none
segment begin allocated size
28b90000 28b91000 28f1aacc 0x389acc(3709644)
9e620000 9e621000 9e96f57c 0x34e57c(3466620)
Large object heap starts at 0x41b91000
segment begin allocated size
41b90000 41b91000 42268f58 0x6d7f58(7176024)
Heap Size: Size: 0xdaffa0 (14352288) bytes.
------------------------------
Heap 15 (1a663990)
generation 0 starts at 0x9faacc7c
generation 1 starts at 0x9faa8ac4
generation 2 starts at 0x29b91000
ephemeral segment allocation context: none
segment begin allocated size
29b90000 29b91000 29cde0e8 0x14d0e8(1364200)
9f620000 9f621000 9fd16c88 0x6f5c88(7298184)
Large object heap starts at 0x42391000
segment begin allocated size
42390000 42391000 42adf6a0 0x74e6a0(7661216)
Heap Size: Size: 0xf91410 (16323600) bytes.
------------------------------
Heap 16 (1a665ec8)
generation 0 starts at 0xc362a47c
generation 1 starts at 0xc3621000
generation 2 starts at 0x2ab91000
ephemeral segment allocation context: none
segment begin allocated size
2ab90000 2ab91000 2afbc464 0x42b464(4371556)
c3620000 c3621000 c3854488 0x233488(2307208)
Large object heap starts at 0x42b91000
segment begin allocated size
42b90000 42b91000 42f635f8 0x3d25f8(4007416)
Heap Size: Size: 0xa30ee4 (10686180) bytes.
------------------------------
Heap 17 (1a668418)
generation 0 starts at 0x94622638
generation 1 starts at 0x94621000
generation 2 starts at 0x2bb91000
ephemeral segment allocation context: none
segment begin allocated size
2bb90000 2bb91000 2bfd1374 0x440374(4457332)
94620000 94621000 948da24c 0x2b924c(2855500)
Large object heap starts at 0x43391000
segment begin allocated size
43390000 43391000 43b7a280 0x7e9280(8295040)
67350000 67351000 6739db20 0x4cb20(314144)
Heap Size: Size: 0xf2f360 (15922016) bytes.
------------------------------
Heap 18 (1a669d20)
generation 0 starts at 0x9a621f68
generation 1 starts at 0x9a621000
generation 2 starts at 0x2cb91000
ephemeral segment allocation context: none
segment begin allocated size
2cb90000 2cb91000 2ce5c30c 0x2cb30c(2929420)
9a620000 9a621000 9a6e597c 0xc497c(805244)
Large object heap starts at 0x43b91000
segment begin allocated size
43b90000 43b91000 43f1f520 0x38e520(3728672)
Heap Size: Size: 0x71e1a8 (7463336) bytes.
------------------------------
Heap 19 (1a66b628)
generation 0 starts at 0x83641300
generation 1 starts at 0x83621000
generation 2 starts at 0x2db91000
ephemeral segment allocation context: none
segment begin allocated size
2db90000 2db91000 2dfaecb8 0x41dcb8(4316344)
83620000 83621000 83855614 0x234614(2311700)
Large object heap starts at 0x44391000
segment begin allocated size
44390000 44391000 44a37488 0x6a6488(6972552)
Heap Size: Size: 0xcf8754 (13600596) bytes.
------------------------------
Heap 20 (1a66cf30)
generation 0 starts at 0x8b621738
generation 1 starts at 0x8b621000
generation 2 starts at 0x2eb91000
ephemeral segment allocation context: none
segment begin allocated size
2eb90000 2eb91000 2ef0c5e4 0x37b5e4(3651044)
8b620000 8b621000 8b94d484 0x32c484(3327108)
Large object heap starts at 0x44b91000
segment begin allocated size
44b90000 44b91000 450100c0 0x47f0c0(4714688)
Heap Size: Size: 0xb26b28 (11692840) bytes.
------------------------------
Heap 21 (1a66e838)
generation 0 starts at 0xf31d3830
generation 1 starts at 0xf31d1000
generation 2 starts at 0x2fb91000
ephemeral segment allocation context: none
segment begin allocated size
2fb90000 2fb91000 2fe8b854 0x2fa854(3123284)
f31d0000 f31d1000 f35a9948 0x3d8948(4032840)
Large object heap starts at 0x45391000
segment begin allocated size
45390000 45391000 458c3008 0x532008(5447688)
Heap Size: Size: 0xc051a4 (12603812) bytes.
------------------------------
Heap 22 (1a670140)
generation 0 starts at 0x9867de74
generation 1 starts at 0x98621000
generation 2 starts at 0x30b91000
ephemeral segment allocation context: none
segment begin allocated size
30b90000 30b91000 3102bbdc 0x49abdc(4828124)
98620000 98621000 988edc84 0x2ccc84(2935940)
Large object heap starts at 0x45b91000
segment begin allocated size
45b90000 45b91000 462adab8 0x71cab8(7457464)
Heap Size: Size: 0xe84318 (15221528) bytes.
------------------------------
Heap 23 (1a671a48)
generation 0 starts at 0xe8c810dc
generation 1 starts at 0xe8c81000
generation 2 starts at 0x31b91000
ephemeral segment allocation context: none
segment begin allocated size
31b90000 31b91000 31de8af0 0x257af0(2456304)
e8c80000 e8c81000 e8f756f8 0x2f46f8(3098360)
Large object heap starts at 0x46391000
segment begin allocated size
46390000 46391000 467d71b0 0x4461b0(4481456)
Heap Size: Size: 0x992398 (10036120) bytes.
------------------------------
Heap 24 (1a673350)
generation 0 starts at 0xa1621544
generation 1 starts at 0xa1621000
generation 2 starts at 0x32b91000
ephemeral segment allocation context: none
segment begin allocated size
32b90000 32b91000 32f74f04 0x3e3f04(4079364)
a1620000 a1621000 a1803858 0x1e2858(1976408)
Large object heap starts at 0x46b91000
segment begin allocated size
46b90000 46b91000 4737fc08 0x7eec08(8317960)
67b90000 67b91000 67d11100 0x180100(1573120)
Heap Size: Size: 0xf35464 (15946852) bytes.
------------------------------
Heap 25 (1a674c58)
generation 0 starts at 0x8c6222b8
generation 1 starts at 0x8c621000
generation 2 starts at 0x33b91000
ephemeral segment allocation context: none
segment begin allocated size
33b90000 33b91000 33edff20 0x34ef20(3469088)
8c620000 8c621000 8ca2c690 0x40b690(4241040)
Large object heap starts at 0x47391000
segment begin allocated size
47390000 47391000 47a011a0 0x6701a0(6750624)
Heap Size: Size: 0xdca750 (14460752) bytes.
------------------------------
Heap 26 (1a676560)
generation 0 starts at 0x9b62150c
generation 1 starts at 0x9b621000
generation 2 starts at 0x34b91000
ephemeral segment allocation context: none
segment begin allocated size
34b90000 34b91000 34fa6200 0x415200(4280832)
9b620000 9b621000 9b8b531c 0x29431c(2704156)
Large object heap starts at 0x47b91000
segment begin allocated size
47b90000 47b91000 48373ec0 0x7e2ec0(8269504)
7aa10000 7aa11000 7ab44168 0x133168(1257832)
Heap Size: Size: 0xfbf544 (16512324) bytes.
------------------------------
Heap 27 (1a677e68)
generation 0 starts at 0x92630b90
generation 1 starts at 0x92621000
generation 2 starts at 0x35b91000
ephemeral segment allocation context: none
segment begin allocated size
35b90000 35b91000 361323f0 0x5a13f0(5903344)
92620000 92621000 929fcd4c 0x3dbd4c(4046156)
Large object heap starts at 0x48391000
segment begin allocated size
48390000 48391000 48b76c48 0x7e5c48(8281160)
f0680000 f0681000 f06f4570 0x73570(472432)
Heap Size: Size: 0x11d62f4 (18703092) bytes.
------------------------------
Heap 28 (1a679770)
generation 0 starts at 0xe1c610dc
generation 1 starts at 0xe1c61000
generation 2 starts at 0x36b91000
ephemeral segment allocation context: none
segment begin allocated size
36b90000 36b91000 37076c64 0x4e5c64(5135460)
e1c60000 e1c61000 e1ed5044 0x274044(2572356)
Large object heap starts at 0x48b91000
segment begin allocated size
48b90000 48b91000 4937c3a8 0x7eb3a8(8303528)
f51d0000 f51d1000 f56afdf8 0x4dedf8(5107192)
Heap Size: Size: 0x1423e48 (21118536) bytes.
------------------------------
Heap 29 (1a67b078)
generation 0 starts at 0xa6621380
generation 1 starts at 0xa6621000
generation 2 starts at 0x37b91000
ephemeral segment allocation context: none
segment begin allocated size
37b90000 37b91000 37ecffc0 0x33efc0(3403712)
a6620000 a6621000 a6873190 0x252190(2433424)
Large object heap starts at 0x49391000
segment begin allocated size
49390000 49391000 49a365c8 0x6a55c8(6968776)
Heap Size: Size: 0xc36718 (12805912) bytes.
------------------------------
Heap 30 (1a67c980)
generation 0 starts at 0xb36238ac
generation 1 starts at 0xb3621000
generation 2 starts at 0x38b91000
ephemeral segment allocation context: none
segment begin allocated size
38b90000 38b91000 38eda4b8 0x3494b8(3445944)
b3620000 b3621000 b36978b8 0x768b8(485560)
Large object heap starts at 0x49b91000
segment begin allocated size
49b90000 49b91000 49ffd360 0x46c360(4637536)
Heap Size: Size: 0x82c0d0 (8569040) bytes.
------------------------------
Heap 31 (1a67e288)
generation 0 starts at 0x79a11784
generation 1 starts at 0x79a11000
generation 2 starts at 0x39b91000
ephemeral segment allocation context: none
segment begin allocated size
39b90000 39b91000 3a35caf0 0x7cbaf0(8174320)
79a10000 79a11000 79ec789c 0x4b689c(4941980)
Large object heap starts at 0x4a391000
segment begin allocated size
4a390000 4a391000 4a94e330 0x5bd330(6017840)
Heap Size: Size: 0x123f6bc (19134140) bytes.
------------------------------
GC Heap Size: Size: 0x1c1341b8 (471024056) bytes.
Based on the output from !vmstat, you are out of memory. There's some mild address space fragmentation, but you only have a total of ~350MB of free memory, so you're really running close to the address space limit. The largest free block is just 6MB, and the CLR allocates virtual memory segments that are at least 16MB in size.
Your total GC heap size is just 470MB (see the last line from the !eeheap -gc output), which means you have other stuff in your process using up address space. Namely, you have >500MB of images (DLLs) and >3GB of memory classified as "Private". This can be a bunch of different things; for example, it can be unmanaged heap allocations.
You can try to further zoom in on the space hog by running !heap -s -h 0 to see if you have large unmanaged heaps in your process. I suggest that once you have a direction (is it an unmanaged heap leak? something else?) to ask another question with your findings. From the information you posted so far, we can conclude it's likely unrelated to what the managed part of your application is doing. Do you have large unmanaged components in your application? There are techniques for analyzing unmanaged memory leaks, such as UMDH or ETW heap allocation tracing.
One final comment: why are you running a 32-bit app on a system with 32 processors? Looks like a server system, and I bet you have more than 4GB of physical memory. If it's at all under your control, try making the move to 64-bit.
I dont seem to have any caching enabled when checking in Opscenter or cfstats. Im running Cassandra 1.1.7 with Solandra on Debian. I have set the required global options in cassandra.yaml:
key_cache_size_in_mb: 800
key_cache_save_period: 14400
row_cache_size_in_mb: 800
row_cache_save_period: 15400
row_cache_provider: SerializingCacheProvider
Column Families were created as follows:
create column family example
with column_type = 'Standard'
and comparator = 'BytesType'
and default_validation_class = 'BytesType'
and key_validation_class = 'BytesType'
and read_repair_chance = 1.0
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'ALL';
Opscenter shows no data available on caching graphs and CFSTATS doesn't show any cache related fields:
Column Family: charsets
SSTable count: 1
Space used (live): 5558
Space used (total): 5558
Number of Keys (estimate): 128
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 61381
Read Latency: 0.123 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Bloom Filter False Postives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 16
Compacted row minimum size: 1917
Compacted row maximum size: 2299
Compacted row mean size: 2299
Any help or suggestions are appreciated.
Sam
The caching stats have been moved from cfstats to info in Cassandra 1.1. If you run nodetool info you should see something like:
Key Cache : size 5552 (bytes), capacity 838860800 (bytes), 38 hits, 47 requests, 0.809 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 838860800 (bytes), 0 hits, 0 requests, NaN recent hit rate, 15400 save period in seconds
This is because there are now global caches, rather than per-CF. It seems that Opscenter needs updating for this change - maybe there is a later version available that will work.