On Linux, FreeBSD and other systems I have valgrind for checking for memory errors like invalid reads and similar. I really love valgrind. Now I have to test code on Solaris/OpenSolaris and can't find a way to get information on invalid reads/writes in an as nice way (or better ;-)) as valgrind there.
When searching for this on the net I find references to libumem, but I get only reports about memory leaks there, not invalid access. What am I missing?
The dbx included with the Sun Studio compilers includes memory access checking support in its "Run Time Checking" feature (the check subcommand). See:
Solaris Studio 12.4 dbx manual: Chapter 9: Using Runtime Checking
Debugging Applications with Sun Studio dbx, dbxtool, and the Thread Analyzer
Leonard Li's Weblog: Runtime Memory Checking
The related "Sun Memory Error Discovery Tool" is also available from
http://cooltools.sunsource.net/discover/
Since version 3.11.0, Valgrind does run on Solaris.
See Release Notes and Supported Platforms.
More precisely, x86/Solaris and amd64/Solaris is now supported.
Support for sparc/Solaris is still in works.
watchmalloc is a quite useful library that can be dynamically loaded for your program (usually no need for recompiling) and then sets watchpoints at all the usually problematic memory locations, like freed areas or after an allocated memory block.
If your program accesses one of these invalid areas it gets a signal and you can inspect it in the debugger.
Depending on the configuration problematic areas can be watched for writes only, or also for reads.
Related
I'm trying to do some debugging on a server on an issue that I suspect is related to a buffer overflow, so I tried to compile my code with -fsanitize=address to enable address sanitizing.
It compiled, and the resulting software runs. However, I'm trying to get a core dump when the address sanitizer detects an error since that is pretty much the only way I can get information out of the system due to the setup.
I am calling the software with ASAN_OPTIONS=abort_on_error=1 prepended on the command line (using a shell script to do that), and have checked that ulimit -c gives unlimited as result, but it just won't produce a core dump.
What am I missing?
This is on an ubuntu 14.04 server with gcc version 4.8.4
EDIT: sysctl kernel.core_pattern gives back kernel.core_pattern = |/usr/share/apport/apport %p %s %c %P. This probably means that apport is enabled (at least in some form). However, I have been able to get proper core files on this system from asserts and SIGFPEs in the software (that is where the suspicion of array overruns comes from).
Let me guess, is this x64 target? Coredumps are disabled there to avoid dumping 16 TB shadow memory (see docs for disable_coredump here for details).
Newer versions of GCC/Clang remove shadow from core by default so that one could do something like
export ASAN_OPTIONS=abort_on_error=1:disable_coredump=0
but I'm afraid 4.8 is too old for this.
As an alternative suggestion, why backtraces are not enough for you? You could use log_path or log_to_syslog to preserve them if you do not have access to programs stderr.
NB: I posted suggestion to enable coredumps on all platforms.
I'm trying to build a program from its source code with VC 11. When the compiler is about to finish, it raises the error mentioned in title of this post.
As I've read here and in other forums, I tried to both close as many programs as possible and enlarge the size of the swap file in Windows... neither works.
I've read about a parameter called \Zm but I don't understand how to use it.
Can you please help me?
Take a look at this documentation which gives possible solutions:
I also had that problem and found the documentation useful. Main points:
If the compiler also issues errors C1076 and C3859, use the /Zm compiler option to lower the memory allocation limit. More heap space
is available to your application if you lower the remaining memory
allocation.
If the /Zm option is already set, try removing it. Heap space might be
exhausted because the memory allocation limit specified in the option
is too high. The compiler uses a default limit if you remove the /Zm
option.
If you are compiling on a 64-bit platform, use the 64-bit compiler toolset. For information, see How to: Enable a 64-Bit Visual C++
Toolset on the Command Line.
On 32-bit Windows, try using the /3GB boot.ini switch.
Increase the size of the Windows swap-file.
Close other running programs.
Eliminate unnecessary include files.
Eliminate unnecessary global variables, for example, by allocating memory dynamically instead of declaring a large array.
Eliminate unused declarations.
Split the current file into smaller files.
I can't tell much about the /Zm parameter, but I had the same issue (compiler is out of heap space).
What has helped me was the /m:4 (4 for the count of your CPUs) parameter so that you can use multiple CPUs for building.
Hope that helps you as well.
Also, if you are running on x64, be sure that the x64 version of "msbuild.exe" and "cl.exe" is beeing used. I had the issue that even when using e.g. the x64 ms powershell, the compiler would still choose the 32-bit version of msbuild.exe (in task manager "msbuild.exe*32", windows 7)
In addition to the other answers here (and in my case), fatal error C1060: compiler is out of heap space can be caused by a syntax error. The following code (in certain circumstances) can cause this error even with correct compiler options -- for instance if you've previously successfully compiled the same program.
r.push_back(e[1];
instead of
r.push_back(e[1]);
It seems to only cause this error rather than the standard error C2143: syntax error: missing ')' before ';' when r and e are of certain types, but it's worth checking any code you've edited recently if the program previously compiled without errors.
We had similar problem: a relativelly simple program (although, full of templates, using Eigen library) persistently failed to compile on one of the computers. All were using MSVC2013 x64, but only one was unable to compile the program due to C1060 error. We tried different compiler flags, setting/unsetting -Zm, but failed to resolve it without modifying code.
Some pointers were, however, given to us, when we switched from x64/x64 (64bit compiler for 64bit resulting executable) version of the compiler to the x86/x86 (32bit compiler for 32bit resulting executable). The x86 compiler gave us exact locations of the problematic parts of the program - calls to template functions receiving heavy templated objects. We have rewritten those to normal functions (build in different object file) and that solved the problem...
VS: Visual Studio 2015
OS: Windows10
If you are using VS2015 as your IDE, maybe there is another solution:
Go to update the VS2015 "Update3" package and everything will work smoothly.
In my case, a main program would not compile is VS 2022 Community Edition (free). It had many include files. By process of elimination, I managed to compile it once I removed any "volatile" modifiers in declarations that had this modifier.
A very strange bug, to say the least!
I got this error when compiling OnnxRuntime with MS Visual C++ 17 2022.
The solution for this issue was to close all other programs and compile using a single thread (in this case, removing the --parallel argument from the build.bat call).
I have been struggling to detect a memory corruption error in our product. The memory detection tools like valgrind only tell the problem at the time of the crash, not when the corruption actually occurs. I have seen while using debug builds that it will check the memory area before and after the block being freed, and show a debug assertion failure saying a heap corruption has occured. So can I rebuild my product in debug mode to capture the error right when it occurs? Will it also catch buffer overruns etc? I could not find any information on the internet about debug builds being targetted towards memory error detection.
You can use as well the 'Page Heap' available on every version of Windows.
You can use gflags that comes along to Debugging Tools for Windows to configure Full Page Heap for your application.
You can then run your application, even in retail mode, under the debugger. The debugger will stop once you encounter a buffer overrun or access to freed memory.
I like very much this tool, because it is built in the OS, an can even be activated on a customer site (gflags only sets registry keys and you can simply send these keys to your customer).
Some people are afraid when we mention (Debugging Tools for Windows). You can use Visual Studio to diagnose the problem. The only thing you need are the PDBs corresponding to your binaries (you can generate them even for release builds).
I am not sure about the debug builds, but for a nice overview of memory corruption tools you might look at http://code.google.com/p/address-sanitizer/wiki/ComparisonOfMemoryTools. It list Valgrind and what it should be able to check for you.
"C++" is not a compiler; vendors make their own according (more or less) to the standard specs.
I only have experience with Microsoft's, and I can tell you that it checks heap corruption by allocating "sentries" around each new or malloc block (in debug mode only, of course) and filling them with a special pattern (was 0xCD when I last used it), and then it checks the guards for every write around that location. If they changed, you'll get a run-time error saying the heap got corrupted.
By the way, buffer overflows are heap corruption.
Edit to add a reference: http://msdn.microsoft.com/en-us/library/8wtf2dfz.aspx
I wrote some program which uses information about (reads via Windows) hardware of the current PC (big program, so I can't post here code) and sometimes my windows 7 crashes, the worst thing is that I have no idea why, and debug doesn't help me, is there any way to receive from windows 7 some kind of log, why it crashed? thanks in advance for any help
The correct (but somewhat ugly) answer:
Go to Computer->Properties, go to 'Advanced System Settings'.
Under startup and recovery, make sure it is set to "Kernel memory dump" and note the location of the dump file (on a completely default install, you are looking at C:\windows\memory.dmp)
You optimally want to install Windows Debugging tools (now in the Windows SDK) as well as setting the MS Symbol store in your symbol settings (http://msdn.microsoft.com/en-us/library/ff552208(v=vs.85).aspx)
Once youv'e done all that, wait for a crash and inspect memory.dmp in the debugger. Usually you will not see the exact crash because your driver vendors don't include symbols, but you will also generally get to see the DLL name that is involved in the crash, which should point you to what driver you are dealing with.
If you are not seeing a specific driver DLL name in the stack, it often indicates to me a hardware failure (like memory or overhead) that needs to be addressed.
MS has a good article here at technet that describes what I mentioned above (but step by step and in greater detail) http://blogs.technet.com/b/askcore/archive/2008/11/01/how-to-debug-kernel-mode-blue-screen-crashes-for-beginners.aspx
You can also look at the event log as someone else noted, but generally the information there is next to useless, beyond the actual kernel message (which can sometimes vaguely indicate whether the problem is driver or something else)
I wonder if there is a good way to find the source code that causes a heap corruption error, given the memory address of the data that was written 'outside' the allocated heap block in Visual Studio;
Dedicated (0008) free list element 26F7F670 is wrong size (dead)
(Trying to write down some notes on how to find memory errors)
Begin with installing windbg:
http://www.microsoft.com/whdc/Devtools/Debugging/default.mspx
Then turn on the pageheap like this:
gflags.exe /p /enable yourexecutable.exe /full
This will insert a non writable page after each heap allocation.
After this launch the executable from inside windbg, any writes outside the heap will now be caught by this debugger. To turn of the pageheap afterwards use this:
gflags.exe /p /disable yourexecutable.exe
More info on how to use the pageheap here.
For Window 10 you could enable the PageHeap option in the GFlags Tool, this tool is included as part of the Debugging Tools for Windows.
The Page Heap options in GFlags lets you select standard heap verification or full-page heap verification. Beware, the full heap verification uses a full page of memory for each allocation so it can cause system memory shortages.
To enable the Page Heap in GFlags:
•To enable standard page heap verification, the standard version will write a pattern at the end of each heap allocation and then examine the pattern when the allocations are freed.
To verify all processes use:
gflags /r +hpa
gflags /k +hpa
for a single process use:
gflags /p /enable ImageFileName
•To enable full page heap verification for one process, this option places an inaccessible page at the end of each allocation so that the program stops immediately if it tries to accesses memory beyond the allocation, this should only be used on a single process due to the heavy memory consumption.
gflags /i ImageFileName +hpa
gflags /p /enable ImageFileName /full
The two commands above are interchangeable.
Note: All page heap settings mentioned above are system wide settings stored in the registry (except /k) and remain effective until you change them. The /k setting is a Kernel flag setting are set for this session and will be lost when Windows shuts down
Another helpful tool is the Application Verifier, but this is not part of the Debugging Tools for Windows, rather it is included in the Windows Software Development Kit (SDK).
Maybe you can try Microsoft's Application Verifier.
It solved a similar problem for me once,by turning on extra checks on heap operations.
In my opinion, the randomness of corrupted address is because the heap can be 'subtly' damaged, and the problem won't show up until something big happens to the heap (like massive allocation/free).
You could set a breakpoint on a write to the memory address. The debugger will then show you the code that writes to the location, but you still need to work out which of the writes are causing the problem.
It's probably too late but if it compiles with gcc and can run on linux you may use valgrind to find the source of the problem (i don't remember the flags, i only used it once with great success).
more info about Gflags and PageHeap(which helped a lot):
http://msdn.microsoft.com/en-us/library/windows/hardware/ff549561%28v=vs.85%29.aspx
I am assuming C++ as the language.
If the error is reproducible and the corrupted address is always the same, you can put a data breakpoint to stop the program when writing at this address.
Make sure all libraries you are linking to are compiled in the same CLR version as the application you are running - all in Release or all in Debug.
When you compile in either Debug and Release you are actually targeting two different versions of the C runtime library. These versions are quite different and they use different strategies for allocating memory and they use different heaps. But the most important thing to know is that they are NOT compatible with one another.
The Release C runtime library allocated memory as expected, whereas the Debug will add extra information, such as guard blocks to track buffer overflow and the location that called the allocation function, and in turn it allocates more memory than the Release.
If your are linking your application to a mix of DLLs which were built in Release and Debug, you will most likely end up trying to delete an object in one CLR which was created in another. This means you will be trying to free more or less memory than what was allocated to the object and this can corrupt the heap.
You should build your application, as well as attach to libraries which are built under the same configuration, either Release or Debug.
This problem can occur especially in modules that are being compiled with different compilers.
There is a way to work around, which I will mention but do not recommend. If for some reason you still need to build in different modes, this work around will allows for all memory to be allocated and freed from the same shared heap. The API GetProcessHeap will allow you to access the shared heap throughout the different modules. By using the HeapAlloc & HeapFree you can allocate and free memory in the shared heap. Note: the HeapAlloc and HeapFree should replace all calls to malloc and free in your application.