I've been trying to be more meticulous lately about memory management in my code. Just for a laugh, I wrote a simple C source file containing only one function, and used valgrind to see if the C compiler itself had any leaks. To my great surprise, it did!
valgrind --leak-check=full --show-reachable=yes gcc -c example.c
...bunch of junk...
==4587== LEAK SUMMARY:
==4587== definitely lost: 4,207 bytes in 60 blocks
==4587== indirectly lost: 56 bytes in 5 blocks
==4587== possibly lost: 27 bytes in 2 blocks
==4587== still reachable: 29,048 bytes in 47 blocks
==4587== suppressed: 0 bytes in 0 blocks
Clang had leaks too, of only 68 bytes though, all of them reachable.
I thought that if your code has memory leaks, you get thrown in solitary confinement for every byte you lose. Have I misunderstood the implications of memory leaks? Are they actually sort of tolerable as long as it's not a long-running program? Is this actually just valgrind being wrong?
Related
Here is the code.
In the golang main function, which in main.go
func main() {
rgc.GetRgcService()
}
where the rgc is in another golang file, named mrgc.go. The code inside is
package rgc
func GetRgcService() (svc *RgcService, err error) {}
The function GetRgcService is a empty function.
However, when I used valgrind to test the memory, I got the following output
==58156== HEAP SUMMARY:
==58156== in use at exit: 1,152 bytes in 4 blocks
==58156== total heap usage: 9 allocs, 5 frees, 1,304 bytes allocated
==58156== 288 bytes in 1 blocks are possibly lost in loss record 4 of 4
==58156== at 0x4A27F63: calloc (vg_replace_malloc.c:593)
==58156== by 0x4010DE1: allocate_dtv (in /home/opt/gcc-4.8.2.bpkg-r2/gcc-4.8.2.bpkg-r2/lib64/ld-2.18.so)
==58156== by 0x40114ED: _dl_allocate_tls (in /home/opt/gcc-4.8.2.bpkg-r2/gcc-4.8.2.bpkg-r2/lib64/ld-2.18.so)
==58156== by 0x4B36DE2: pthread_create##GLIBC_2.2.5 (in /home/opt/gcc-4.8.2.bpkg-r2/gcc-4.8.2.bpkg-r2/lib64/libpthread-2.18.so)
==58156== by 0x4B2937: _cgo_sys_thread_start (gcc_linux_amd64.c:75)
==58156== by 0x45506C: runtime.asmcgocall (/home/map/.jumbo/lib/go/src/runtime/asm_amd64.s:612)
==58156== by 0x50619F: ??? (in /home/users/zhanghuaizhi/ttt.38)
==58156== by 0xC7FFFFFFFF: ???
==58156== by 0xC820067FFF: ???
==58156== by 0x42D69B: runtime.allocm (/home/map/.jumbo/lib/go/src/runtime/proc.go:1260)
==58156== by 0x42DD3A: runtime.newm (/home/map/.jumbo/lib/go/src/runtime/proc.go:1510)
==58156== by 0x42E071: runtime.startm (/home/map/.jumbo/lib/go/src/runtime/proc.go:1583)
==58156==
==58156== LEAK SUMMARY:
==58156== definitely lost: 0 bytes in 0 blocks
==58156== indirectly lost: 0 bytes in 0 blocks
==58156== possibly lost: 1,152 bytes in 4 blocks
==58156== still reachable: 0 bytes in 0 blocks
==58156== suppressed: 0 bytes in 0 blocks
How can I free these memory? Since I need to used this function to do a lot of process. It causes lots of memory leaks, which can not be freed
Nothing was leaked. The memory is still reachable and it's quite common to not free things on exit, it just takes unnecessary time and the OS will deal with it anyway.
This is memory allocated to thread local storage to a thread that's still running, so it would be incorrect to free it. A better question would be "how do I stop this thread?", to which an answer is: you don't, the Go runtime deals with it. It is quite common to not stop threads at exit, it just takes unnecessary time and the OS will deal with it anyway.
It has nothing to do with your code and your function call, it's something that Go runtime allocates for itself.
Go is a garbage collected language and using valgrind on it will not tell you much. It will neither detect real memory leaks nor will it understand which memory is still in use.
You can manually run garbage collection:
https://golang.org/pkg/runtime/#GC
I think that would free up the memory, but as others have said, that memory will get freed eventually when the runtime's scheduled garbage collector runs.
We are running a mixed mode process (managed + unmanaged) on Win 7 64 bit.
Our process is using up too much memory (especially VM). Based on our analysis, the majority of the memory is used by a big native heap. Our theory is that the LFH is saving too many free blocks in committed memory, for future allocations. They sum to about 1.2 GB while our actual allocated native memory is only at most 0.6 GB.
These numbers are from a test run of the process. In production it sometimes exceeded 10 GB of VM - with maybe 6 GB unaccounted for by known allocations.
We'd like to know if this theory of excessive committed-but-free-for-allocation segments is true, and how this waste can be reduced.
Here's the details of our analysis.
First we needed to figure out what's allocated and rule out memory leaks. We ran the excellent Heap Inspector by Jelle van der Beek and we ruled out a leak and established that the known allocations are at most 0.6 deci-GB.
We took a full memory dump and opened in WinDbg.
Ran !heap -stat
It reports a big native heap with 1.83 deci-GB committed memory. Much more than the sum of our allocations!
_HEAP 000000001b480000
Segments 00000078
Reserved bytes 0000000072980000
Committed bytes 000000006d597000
VirtAllocBlocks 0000001e
VirtAlloc bytes 0000000eb7a60118
Then we ran !heap -stat -h 0000001b480000
heap # 000000001b480000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
c0000 12 - d80000 (10.54)
b0000 d - 8f0000 (6.98)
e0000 a - 8c0000 (6.83)
...
If we add up all the 20 reported items, they add up to 85 deci-MB - much less than the 1.79 deci-GB we're looking for.
We ran !heap -h 1b480000
...
Flags: 00001002
ForceFlags: 00000000
Granularity: 16 bytes
Segment Reserve: 72a70000
Segment Commit: 00002000
DeCommit Block Thres: 00000400
DeCommit Total Thres: 00001000
Total Free Size: 013b60f1
Max. Allocation Size: 000007fffffdefff
Lock Variable at: 000000001b480208
Next TagIndex: 0000
Maximum TagIndex: 0000
Tag Entries: 00000000
PsuedoTag Entries: 00000000
Virtual Alloc List: 1b480118
Unable to read nt!_HEAP_VIRTUAL_ALLOC_ENTRY structure at 000000002acf0000
Uncommitted ranges: 1b4800f8
FreeList[ 00 ] at 000000001b480158: 00000000be940080 . 0000000085828c60 (9451 blocks)
...
When adding up up all the segment sizes in the report, we get:
Total Size = 1.83 deci-GB
Segments Marked Busy Size = 1.50 deci-GB
Segments Marked Busy and Internal Size = 1.37 deci-GB
So all the committed bytes in this report do add up to the total commit size. We grouped on block size and the most heavy allocations come from blocks of size 0x3fff0. These don't correspond to allocations that we know of. There were also mystery blocks of other sizes.
We ran !heap -p -all. This reports the LFH internal segments but we don't understand it fully. Those 3fff0 sized blocks in the previous report appear in the LFH report with an asterisk mark and are sometimes Busy and sometimes Free. Then inside them we see many smaller free blocks.
We guess these free blocks are legitimate. They are committed VM that the LFH reserves for future allocations. But why is their total size so much greater than sum of memory allocations, and can this be reduced?
Well, I can sort of answer my own question.
We had been doing lots and lots of tiny allocations and deallocations in our program. There was no leak, but it seems this somehow created a fragmentation of some sort. After consolidating and eliminating most of these allocations our software is running much better and using less peak memory. It is still a mystery why the peak committed memory was so much higher than the peak actually-used memory.
I am developing SW for embedded Linux and i am suffering system hangs because OOM Killer appears from time to time. Before going beyond i would like to solve some confusing issues about how Linux Kernel allocate dynamic memory assuming /proc/sys/vm/overcommit_memory has 0 and /proc/sys/vm/min_free_kbytes has 712, and no swap.
Supposing embedded Linux currently physical memory available is 5MB (5MB of free memory and there is not usable cached or buffered memory available) if i write this piece of code:
.....
#define MEGABYTE 1024*1024
.....
.....
void *ptr = NULL;
ptr = (void *) malloc(6*MEGABYTE); //Preserving 6MB
if (!prt)
exit(1);
memset(ptr, 1, MEGABYTE);
.....
I would like to know if when memset call is committed the kernel will try to allocate ~6MB or ~1MB (or min_free_kbytes multiple) in the physical memory space.
Right now there is about 9MB in my embedded device which has 32MB RAM. I check it by doing
# echo 3 > /proc/sys/vm/drop_caches
# free
total used free shared buffers
Mem: 23732 14184 9548 0 220
Swap: 0 0 0
Total: 23732 14184 9548
Forgetting last piece of C code, i would like to know if its possible that oom killer appears when for instance free memory is about >6MB.
I want to know if the system is out of memory when oom appears, so i think i have two options:
See VmRSS entries in /proc/pid/status of suspicious process.
Set /proc/sys/vm/overcommit_memory = 2 and /proc/sys/vm/overcommit_memory = 75 and see if there is any process requiring more of physical memory available.
I think you can read this document. Is provides you three small C programs that you can use to understand what happens with the different possible values of /proc/sys/vm/overcommit_memory .
I recently switched from ruby 1.8.7 (p371) to ruby 1.9.3 (p429), and observed high memory leak(?). My long-running scripts eventually all get killed due to used up memory. Tried to use some profiler (rubyprof, memprof, bleakhouse,etc) to track down but seems they don't have good support for 1.9.3.
Just used valgrind to track the memory usage from C level and observed high leakage, I run this sample codes:
A = ['na', 'ns', 'n/a.jpg']
b = 'N/A.jpg'
if A.include?(b.downcase)
puts 'yes'
end
Very simple. In ruby 1.8.7, I tried valgrind --partial-loads-ok=yes --undef-value-errors=no --leak-check=full ruby test.rb, and it gave the result:
LEAK SUMMARY:
==28315== definitely lost: 0 bytes in 0 blocks
==28315== indirectly lost: 0 bytes in 0 blocks
==28315== possibly lost: 16,464 bytes in 431 blocks
==28315== still reachable: 15,103,747 bytes in 110,826 blocks
==28315== suppressed: 0 bytes in 0 blocks
While in 1.9.3, it gave the following result:
LEAK SUMMARY:
==28228== definitely lost: 2,028,264 bytes in 9,246 blocks
==28228== indirectly lost: 1,221,720 bytes in 23,812 blocks
==28228== possibly lost: 21,251 bytes in 432 blocks
==28228== still reachable: 754,268 bytes in 15,849 blocks
==28228== suppressed: 0 bytes in 0 blocks
and I observed leak in rb_longjmp, rb_is_eq_eval, rb_class_new_instance, rb_ary_each.... methods. Is that mean ruby 1.9.3 have a leakage? Any fix to that?
I have seen some of the post regarding valgrind but not single post helped me in understanding interpretation of valgrind output.
I ran two programs with valgrind (Both having memory leaks)
Sample Output for Test 1
==20422== LEAK SUMMARY:
==20422== definitely lost: 448 bytes in 3 blocks
==20422== indirectly lost: 786,460 bytes in 1 blocks
==20422== possibly lost: 1,576,052 bytes in 46 blocks
==20422== still reachable: 1,077,107 bytes in 2,333 blocks
==20422== suppressed: 0 bytes in 0 blocks
==20422== Rerun with --leak-check=full to see details of leaked memory
==20422==
==20422== For counts of detected and suppressed errors, rerun with: -v
==20422== ERROR SUMMARY: 98307 errors from 5 contexts (suppressed: 2 from 2)
Killed
Output for Test 2
==20875== HEAP SUMMARY:
==20875== in use at exit: 1,059,198 bytes in 2,047 blocks
==20875== total heap usage: 3,019 allocs, 972 frees, 4,496,090 bytes allocated
==20875==
==20875== LEAK SUMMARY:
==20875== definitely lost: 328 bytes in 2 blocks
==20875== indirectly lost: 0 bytes in 0 blocks
==20875== possibly lost: 1,600 bytes in 5 blocks
==20875== still reachable: 1,057,270 bytes in 2,040 blocks
==20875== suppressed: 0 bytes in 0 blocks
==20875== Rerun with --leak-check=full to see details of leaked memory
==20875==
==20875== For counts of detected and suppressed errors, rerun with: -v
==20875== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
If you look at the error summary, One output says there are errors and other says there are no errors (eventually both have leaks)
My question is - - what is counted as error in valgrind? (valgrind manual didn't helped me)
Memory leaks are not considered errors, they don't effect the logic of the program.
Errors are things such as invalid reads and writes.
Updated as result of comment: The invalid reads and writes in test 1 are for 5 different areas of memory which are accessed 98307 times.
Looking at the leaks, the large indirect losses in test 1 may indicated a linked data structure where only the root has been deleted.
The leaks in test 2 aren't too bad. As valgrind suggests rerun with --leak-check=full which should indicate which bit of code is causing the problem.
A full explanation of the errors can be found here