Rational Functional Tester vb6 - Stackhash Crash on Win7 - vb6

Ive got an issue using RFT with VB6 on Win7 (32,64) bit. This isnt is not present on Windows XP.
I am unable to highlight objects, or call verification points as the VB6 application crashes with a stackhash error. I have done weeks of research and have tried various things(DEP, Compatibility mode, clean boots etc) but am thus far unable to isolate the issue.
It seems that certain properties, these can vary, are not available once the program has been compiled. Running the application through VB6 IDE in debug mode does not result in a crash. Once compiled though, the crash occurs constantly.
I have used Windbg to debug the error to see what happens, and can see that there is a call to unallocated memory, resulting in heap corruption. What causes this? Is there anyway around this?
Any help would be appreciated :)

Related

Windows: problem closing 32-bit consolle application

I am working on an application that when compiled in debug mode for Win32 (on a x64 machine), just before exiting, throws this exception:
From what I was able to reconstruct, it seems that the offending line of code is this, in the xmemroy0.h header:
// If the following asserts, it likely means that we are performing
// an aligned delete on memory coming from an unaligned allocation.
_STL_ASSERT(_Ptr_user[-2] == _Big_allocation_sentinel, "invalid argument");
It seems that this happens when a dll of mine (which is part of the application) is unloaded by the SO. It doesn't seem to be a directly a problem of my software, but I fear I could be the indirect cause of the problem by not handling correctly the dll boilerplate code. For example I am doing nothing special to handle attachments/detachments in the dll. What seems wired to me is that the x64 version seems to work just fine.
Any clue from a senior system programmer that could point me in the right direction would be highly appreciated. I don't really know how to debug this, I am not an expert of the Windows internals.
Sincerely,

Code doesn't work in debug

i coded a big project in c++ that runs when I open it in Debug or Release Mode, but when i open it without Debugging (ctrl + f5) it crashs after 5 seconds. It just doesn't reply anymore and is tagged as inactive in taskmanager. I tried to analyse the error with the windows debugger tools and application verfier, but i found nothing. Even when I set the "_NO_DEBUG_HEAP=1", the error doesn't occur, it just happens when I start it with ctrl + f5 or outside from visual studio. I'm not even sure if the _NO_DEBUG_HEAP works... Anyone have an idea what could be wrong ?
Any number of things may cause this.
Based on my passed experience, I suggest tackling this by removing code until the behaviour disappears... then determining why removing the code in question corrected the behaviour. (the removed code may not have been the cause, only a catalyst for the symptom)
If I was tasked with taking a quick glance to spot the error, I would think to look for buffer overrun related issues. This type of error has more safeguards in debug than executing a final build. That's just a stab in the dark based on buffer overruns being a common issue that kinda fits the symptom.
-dm
Common things to check:
Are you sure your application is setting any memory used to zero,
if it is assuming the memory has to be zeroed out?
Visual Studio might be initializing the memory given to the
application.
Is all the memory allocated being deallocated?
There are no un-initialized pointers being used?
There are no un-initialized variables being used?
64 bit exe crashing outside visual studio but working inside visual studio
Does the problem only happen on one computer?
You can use check_heap to check the validity of the heap
in the program:
Program crashes only in Release mode outside debugger
You can use the Windows debugging tools to show heap
corruption - there is a tool called "gflags" that comes with the
Microsoft "Standalone Debugging Tools for Windows"
https://msdn.microsoft.com/en-us/windows/hardware/hh852365
This URL shows how to run gflags against your .EXE:
Visual Studio - how to find source of heap corruption errors

Troubleshooting VB6 App Crash after XP to Win7 Upgrade

I have a VB6 application that I provide support for. This application works on both Windows XP and Windows 7. Some users were migrated from Windows XP to Windows 7 using the User State Migration tool. These users now receive a generic "Application has crashed" Windows error message when they open certain screens (forms) in the application. My assumption is that there is a missing dll/ocx reference, but I'm having trouble tracking it down.
I've tried many/varied troubleshooting techniques:
Full uninstall and reinstall of my application
Manually re-registering all dll's and ocx's that I know are used
Running Process Monitor on a broken computer and a working computer to compare what dll's and ocx's are accessed. The answer might be here but even after filtering out most of the background noise the amount of data is overwhelming. At a minimum I reviewed all of the calls right before it crashes and all of the calls that were not successful. All of the non-successful calls match between working and non-working.
Installed the Windows Debugger Tools and captured a crash dump. Analyzed the crash dump with DebugDiag. DebugDiag says the exception is in msvbvm60.dll. I tried building a PDB file for my exe and loading it in DebugDiag to get more detail about where the exception is occuring but DebugDiag doesn't want to accept the PDB (might be doing something wrong here, but it just seems to ignore it. This same PDB file works fine when I do remote debugging, however.)
I recompiled my VB6 program without any optimizations in PCode. I've read online that sometimes building in PCode, while bad for performance, will tell you the real exception.
Used the above created PDB file to remote debug the VB6 application. The debugger says that the application crashes after the new window has been created, on a line that sets MousePointer = vbHourGlass... To me it seems unlikely that this is the real cause of the error. There are at least 20 other locations in the program where this same line is called and all work fine.
(Forgot about this one)
Used Dependency Walker and profiled the application on both a working and non-working computer. All errors found by dependency walker were the same between the two computers. There were no additional dependencies found on a working computer, and all missing dependencies on the non-working computer were also missing on the working one.
None of these actions changed my error message or showed me what the error is (unless it really is the mouse cursor issue)... There are no entries in the Windows Event Log related to the app crash.
The non-working and working computers all have the same base Windows 7 image, the only difference is whatever is being changed by USMT, which further convinces me that this is some kind of quirky configuration change or a missing dll/ocx or perhaps an unregistered dll/ocx.
Any ideas or thoughts on how I can track down the root cause of the issue would be greatly appreciated.
Update 1 - Response to questions
#MarkHall I have tried running it as admin, though not with UAC off. The application runs fine on a Windows 7 box as a non-admin with full UAC. Windows XP was 32-bit, Windows 7 is 64-bit, but again it works just fine on a like for like box where the user was not migrated from Windows XP.
#Beaner It's possible that it stores settings somewhere that have been corrupted, but the remote debugging leads me to think that it's more likely something else since it seems to die on a step related to the UI, which then makes me think it's probably a missing dll/ocx reference.
#Bob77 The application is installed into Program Files (x86). While many of the libraries do reside in the same folder, they are all registered.
Peter, often I've noticed that the debugger will indicate a line of code that is actually incorrect, depending on WHERE in the actual assembly language the fault occurs. You should look REAL close around your statement that sets the cursor to vbHourGlass. Your exception is PROBABLY happening BEFORE that line of code, but that line is what the debugger thinks is the actual faulted line of code.
Since you said it happens when a window OPENS, I'd look real close at any ocx's you may have referenced on the form, but perhaps NOT actually being used, or called. You might have one there that you don't intend to be there, that could be causing security issues, or something on Win7? Edit the .frm file by hand if you have to, and look at all the GUIDs the form references.
It is possible that one machine is using PER-USER registration, and the other is using PER-SYSTEM registration?? I don't know...
I would take a much closer look at the form that you are trying to open, and be VERY cautious of everything you are doing in the form load events, and so on. This sounds like it could be something as stupid as Windows Aero being enabled on one system, and not another, or some other sort of "Theme" setting that is throwing the VB Form Rendering routine into a hissyfit... Perhaps even something as stupid as a transparent color index in the icon you selected for that from?
If you are still developing this app, (or at least maintaining it), create an entirely NEW form, and re-create all the controls, etc, on the form (resist the temptation to copy/paste them from the old one...), and then see if THAT does the trick. Then, copy all the event code to the new form one event at a time, with at LEAST enough event code to make the form function, even if it's just a "dead form", that loads no data, or whatever the form is supposed to do. Check and debug after each change, and you WILL find it eventually. Of course, make sure you isolate one of the defunct systems to have a platform that you can duplicate the issue on, or then it's just guessing. I find that using something like Acronis w/ Universal Restore is a great option to then take the image file into a good HV, like VirtualBox, and then restore that image as a VM, so you can debug without interfering with your actual users. This sounds like a lot of work, but then again, so is re-writing an application that already exists, right? :)
Failing THAT... /* and */ are your friends!! (Well, we're dealing with VB, so ' would be your best friend! heh... But I'd start commenting out all the code on the form until that sucker opens. Then once it opens, start putting one line back at a time, and re-running it... That's called "VooDoo Debugging", but sometimes, you gotta do what you gotta do...
THANKS A LOT PETER! :) Now you got ME so involved in this, I feel like I'M the one debugging this sucker! Like if it was MY code I was trying to fix! :)
Let me know if any of this helps... I am actually quite interested in what you discover.

HeapAlloc causes excpetion - Windows 7 x64

Before some time i coded my own little db editor program, i was coding it from the zero using Win API's so its not very small project.
It was working fine on all OS till now, i have Win 7 x64 with all latest updates and my application is crashing with 0xC000005 exception because of some of the Heap functions(HeapAlloc or HeapFree, i use nothing else), i tried replacing HeapAlloc & HeapFree with VirtualAlloc and VirtualFree and it was all fine, but i dont want to use the virtual memory....
Something else, i tried to attach with debugger to trace the problem, but when i attach debugger its not crashing, then i tried to display MessageBox to trace where it crashes, but when i display MessageBox its not crashing too....
My application is running as 32bit process.
Coded in C.
Anyone had similar problem ?
Firstly, both HeapAlloc and VirtualAlloc allocate virtual memory.
My guess as to what is happening is that you are writing past the boundary of the allocated memory. The reason why this does not work with HeapAlloc is that it allocates exactly the amount of memory you request. With VirtualAlloc, the size returned is the requested size rounded up to the next page boundary. In your case, this gave a bit more leeway (even though your code is still doing the wrong thing).
In terms of why it has been working so far: you just got lucky. Look carefully at the code accessing the allocated memory and if you get stuck, post the relevant part up here. If the debugger isn't helping and the bug is easily reproducible, just comment out parts of the code until you locate the line causing the crash.
Have you attached it to Debug version of your application? If the problem does not appear in debug version then you should check what warnings (on highest level) generate your code, maybe you will find some uninitialized variables. If nothing here, then you might use some static analysis tool to help with finding bugs - like PVS-Studio http://www.viva64.com/.
You can also compile Release version with debugging information enabled, this way when problem arrises you should be able to attach to your application with debugger and see callstack with function names. To make it easier to debug, disable code optimizations.
You can also try gflags from windows debugger tools, this program will trigger breakpoint each time you write outside of buffer boundary. This is really helpfull tool because not all buffer overruns end up with exceptions. Use it on application with debugging information enabled, and preferably with code optimizations off. To enable gflags for your app use:
gflags /p /enable myapp.exe /full
(http://msdn.microsoft.com/en-us/library/windows/hardware/ff543097%28v=vs.85%29.aspx)

What is the "Cannot set allocations" error, who emits it and what can I do about it?

We've been plagued for several years by occasional reports from customers about a non-descript error message "Cannot set allocations" that appears on startup of our app. We have never been able to reproduce the problem in our own test environments so far. I have now run out of ideas for attempting to track this down. Here's a collection of observations that have accumulated over time:
Error message text reads "Cannot set allocations" (note absence of punctuation).
The window title simply reads "Error" (or the localized equivalent).
The "Cannot set allocations" text is always in English regardless of OS locale.
I have so far not been able to locate the DLL or EXE containing the message text.
Google is chock full of reports of this error for a variety of different products - but no solutions.
The only unifying aspect between the affected products I could make out so far was that they all appear to come in the form of DLLs that load into third-party processes (such as addins for Visual Studio or Windows Explorer shell extensions).
Our app is actually a shareware COM-addin for MS Outlook, written in Delphi (i.e. native code - no .NET).
The prime suspect in our case is the third-party licensing wrapper that we're using which decrypts and uncompresses our DLL into memory on the fly. Obviously I couldn't simply give an unprotected version of our app to the affected customers to verify this suspicion. Maybe the other vendors that this has been reported against are using similar products.
Debug versions of the protection wrapper supplied to us by the licensing vendor yielded no results: The log files looked exactly the same as from sessions where the error did not occur. Apparently the "inner" DLL gets decrypted and uncompressed all right but for some reason still fails to get loaded by the host process.
By creating an unprotected "loader" DLL we have been able to pinpoint the occurrence of the error somewhere behind the LoadLibrary call that is supposed to load our DLL into memory.
Extensive logging and global exception hooks in our own code (both the unprotected loader and the protected "core"-DLL) yielded no results at all. The error is obviously raised somewhere else.
The problem described in this earlier question of mine was very probably prompted by the same issue. This was before we created the unprotected loader stub.
The error only occurs at about 1-2% of our customers - whereas typically all installations at any affected customer's site are affected the same.
Sometimes the error goes away after we release a new version but often it will come back again after a couple of weeks or months.
Once the error has started to occur on a machine it does so consistently.
The error never occurs while connected to the affected machine via remote access (e.g. VNC, RDP, TeamViewer, etc.) and none of the affected customers are within travel distance from us so all we have to go by is log files and "eye-witness reports".
One customer reported that the error message dialog apparently was non-modal, i.e. he was able to simply move the dialog box to the side and continue working with the application (minus the functionality that our DLL would have provided). Not sure whether this is universally true in all other occurrences, too.
In some cases customers have been able to permanently rid themselves of the error by disabling or uninstalling other addins from other vendors that were sharing the host application with our own product.
The error has so far been observed on Windows XP, Vista and 7.
During the last few weeks we had a surge of reports from Outlook 2003 / Windows 7 users. Could the situation have been made worse by a recent Windows/Office-update?
Does anyone have any experience with this error at all?
Or any more ideas for investigating this?
I have only recently had this happen, which prompted my search and I ended up here. I can tell you that with me for sure it is in windows 7 home premium BUT ONLY WITH IE9 (which I hate by the way) it reduces the user back to the dummy stage and assumes that we have to be repeated flagged about everything.It will keep asking you if you want to disable add ons to speed up load times but usually on things that aren't really the things slowing the browser down in the first place,it is there is too much garbage loading in the first place.But back to the "Cannot set allocations", I for one have never expirianced it in any other browser which is not to say it doesn't happen with them.
This is going to be pure guess-work, but it sounds like maybe your third-party licensing software is trying to load your DLL at a particular location in memory, which - on these failing systems - happens to already be occupied by something else, perhaps a global hook DLL.
If you have a customer who is willing to work with you a bit, it might shed some light on the situation to get a crash dump (e.g., with ADPlus or maybe simpler with Sysinternals' ProcDump) when the error message is showing. That would show what modules are loaded and possibly the callstack (if it is from a message box at the time of the error as opposed to one that is catching an exception after the problem).
I also have experienced the "Cannot set allocations" issue. Royal pain. I had disabled Java, since I did not seem to need it, I used add/remove programs to remove Java from my system. Then I stated to get those errors. I have reinstalled, but disabled Java in IE explorer. Now I do not get the error anymore. Not a programmer, don't know why this happened. Maybe a clue for someone.
Win 7 - 64bit OS IE Explorer 10. Hope this helps someone figure this out. John
I've seen this happen. In my case a global hook dll created a large memory file mapping, perhaps to the memory the licensing dll was counting on.
I see "Cannot set Allocations" when I open google chrome only. Also after that, chrome closes with a msg saying "Whoa chrome has crashed..."
Still no solution :(
Also not a programmer, but it always happens when I open Chrome. It opens second window with error message 'cannot set allocation'. I usually close it and go on my way. if i don't it usually causes a crash. doesnt happen on any other browsers.

Resources