Why does this simple DLL crash before WM_ACTIVATEAPP arrives? - windows

Question

The minimal DLL below, which uses only the Win32 API, tries to do nothing more than create an MDI frame/client window and one child window, and destroy the frame window when the DLL unloads. The DLL crashes on Windows XP with an exception upon executing an INT x2B instruction in USER32.
For testing, the DLL is simply invoked by a one-line application calling LoadLibrary('badcode.dll').
The crash happens inside the final "DestroyWindow(framewindowhandle)" just before the DLL finishes, after FrameWindowProc receives WM_ACTIVATE but before it receives WM_ACTIVEAPP.
The DLL code has been trimmed down from a much larger original as much as possible to isolate the bug. Although not destroying the frame window also makes the current crash go away, about 12 years ago it was determined that tools like Visual Basic running on NT would crash unless "DestroyWindow(framewindowhandle)" was called before the DLL was unloaded. Just recently, however, a new small program written to test some of the DLL entrypoints was suddenly found to be crashing on XP as described above.
Although written in Delphi 6, the code only relies on the vanilla Win32 API.
library badcode; // works if rewritten as a program instead of DLL
{$R *.RES} // removing this avoids crash
uses windows, messages; // only win32 calls are made
// 3 MDI window handles
var framewindowhandle, clientwindowhandle, childwindowhandle: hwnd;
function framewindowproc(windowhandle: hwnd; message: word; wparam, lparam: longint): longint; stdcall;
var ccs: tclientcreatestruct;
begin // frame window has received a message
if message = WM_CREATE then
begin // create the client window
ccs.hwindowmenu := 0; ccs.idfirstchild := 0;
clientwindowhandle := createwindow('MDICLIENT', '', ws_child + ws_clipchildren + ws_visible, 10, 10, 50, 50, windowhandle, 0, hinstance, #ccs);
result := 0; // we handled the message
end
else // do default handling
result := defframeproc(windowhandle, clientwindowhandle, message, wparam, lparam);
end;
function childwindowproc(windowhandle: hwnd; message: word; wparam, lparam: longint): longint; stdcall;
begin // child window has received a message, do default handling
result := defmdichildproc(windowhandle, message, wparam, lparam);
end;
procedure DLLHandler(reason: integer);
begin
if reason = DLL_PROCESS_DETACH then // unloading dll
DestroyWindow(framewindowhandle); // causes the crash, never returns
end;
var wc: twndclass; mcs: tmdicreatestruct;
begin // DLL loading time
DLLProc := #DLLHandler; // so we can detect unload
wc.hinstance := hinstance;
wc.lpfnwndproc := #framewindowproc;
wc.style := 0; wc.cbclsextra := 0; wc.cbwndextra := 0;
wc.hicon := loadicon(0, IDI_ASTERISK);
wc.hcursor := loadcursor(0, IDC_ARROW);
wc.hbrbackground := 0;
wc.lpszmenuname := 'MENUBAR'; // changing to '' avoids the crash
wc.lpszclassname := 'BAD';
registerclass(wc); // register the frame window
wc.lpfnwndproc := #childwindowproc;
wc.lpszmenuname := '';
wc.lpszclassname := 'DATA';
registerclass(wc); // register the child window
framewindowhandle := createwindow('BAD', 'frame', WS_OVERLAPPEDWINDOW + WS_CLIPCHILDREN, 100, 100, 400, 600, 0, 0, hinstance, nil);
mcs.szclass := 'DATA'; mcs.sztitle := 'child'; mcs.howner := hinstance;
mcs.x := 50; mcs.y := 50; mcs.cx := 50; mcs.cy := 50; mcs.style := WS_MINIMIZE; // changing the style avoids the crash
childwindowhandle := sendmessage(clientwindowhandle, WM_MDICREATE, 0, longint(#mcs));
sendmessage(clientwindowhandle, WM_MDIRESTORE, childwindowhandle, 0); // skipping this avoids the crash
end.

Answer 1

Using the excellent dependencywalker tool, I discovered some old scanner software on my machine had configured USER32 to hook in an OCR-related DLL upon the execution of any program, and that DLL was making some questionable-looking calls, including being loaded twice for some reason. Uninstalling the scanner software made the crash go away and all O/S DLL loading/unloading look much more reasonable. Nevertheless, I'll be modifying my DLL to do nothing during attach/detach, and include new entrypoints for starting/stopping.