Windows API: Simple window does not redraw correctly - windows

Question

I'm working on a Lua binding for Windows API. So far I've been able to create a basic window with a list box control:
require('Alien')
package.path = 'libs\\?.lua;libs\\?\\init.lua;' .. package.path
local function printf(...) io.write(string.format(...)) end
Windows = require('Windows')
local W = Windows
print("loaded OK")
print("Command line=", W.GetCommandLine())
local windowClassName = "testWindow"
local windowClass, window
local hInstance = assert(W.GetModuleHandle(nil))
assert(W.WM_CREATE)
assert(W.WM_CLOSE)
assert(W.WM_DESTROY)
assert(W.DefWindowProc)
assert(W.DestroyWindow)
assert(W.PostQuitMessage)
local msgID = {}
for k, v in pairs(Windows) do
if k:match('^WM_') then msgID[v] = k end
end
local function wndProc(hWnd, msg, wParam, lParam)
local id = msgID[msg] or ('%08X'):format(msg)
printf('wndProc(%08X, %20s, %08X, %08X\n',
hWnd,id, wParam, lParam)
if msg == W.WM_CREATE then
printf("window %08X create\n", hWnd)
elseif msg == W.WM_CLOSE then W.DestroyWindow(hWnd)
elseif msg == W.WM_DESTROY then W.PostQuitMessage(0)
else return W.DefWindowProc(hWnd, msg, wParam, lParam)
end
return 0
end
local wndProcCB = alien.callback(wndProc,
assert(W.LRESULT), assert(W.HWND),
assert(W.UINT), assert(W.WPARAM), assert(W.LPARAM))
windowClass = assert(W.WNDCLASSEX:new {
cbSize = assert(W.WNDCLASSEX.size),
style = bit.bor(W.CS_HREDRAW, W.CS_VREDRAW),
lpfnWndProc = assert(wndProcCB),
cbClsExtra = 0,
cbWndExtra = 0,
hInstance = assert(hInstance),
hIcon = nil,
hCursor = nil,
hbrBackground = assert(W.COLOR_APPWORKSPACE)+1,
lpszMenuName = nil,
lpszClassName = assert(windowClassName),
hIconSm = nil,
})
assert(W.RegisterClassEx(windowClass))
local dwExStyle = bit.bor(
W.WS_EX_TOOLWINDOW,
W.WS_EX_OVERLAPPEDWINDOW)
local dwStyle = bit.bor(
W.WS_OVERLAPPEDWINDOW,
W.WS_CLIPSIBLINGS,
W.WS_CLIPCHILDREN)
window = assert(W.CreateWindowEx(
assert(dwExStyle), --dwExStyle
assert(windowClassName), --lpClassName
"Test Window", --lpWindowName
assert(dwStyle), --dwStyle
assert(W.CW_USEDEFAULT), --x
assert(W.CW_USEDEFAULT), --y
420, 314, --width, height
nil, nil, assert(hInstance), nil)) --hWndParent, hMenu, hInstance, lpParam
local SWP_SHOW_ONLY = bit.bor(
W.SWP_ASYNCWINDOWPOS,
W.SWP_SHOWWINDOW,
W.SWP_NOACTIVATE,
W.SWP_NOMOVE,
W.SWP_NOSIZE,
W.SWP_NOZORDER)
W.SetWindowPos(assert(window), nil, 0, 0, 0, 0, assert(SWP_SHOW_ONLY))
local msg = assert(W.MSG:new())
while W.GetMessage(msg, 0, 0, 0) do
W.TranslateMessage(msg)
W.DispatchMessage(msg)
end
It creates and displays a window, but the window doesn't redraw correctly. From the console output I can see lots of WM_PAINT, WM_ERASEBKGND, WM_NCPAINT etc, which I pass on to DefWindowProc, but it seems to be not handling them.
Example screenshots:
When the window first appears, it either takes the image of whatever was behind it (as in this case) or appears solid gray. As it's dragged around, it keeps that image and anything that drags over it leaves behind a ghost image. (Can be seen on the titlebar, where I have a small white button following the active window around.) If I drag it off screen a bit, it starts to get even more glitchy but never successfully repaints itself.
I've tried various methods of handling WM_PAINT, WM_ERASEBKGND, etc but none have had any effect. Examples don't show these messages being handled either. Am I doing something wrong, or is there something else I need to be doing?

Answer 1

Well, I found the issue (I hope) with the help of Rohitab API Monitor. It was indeed a problem with the Lua binding: it had DefWindowProc declared as taking four int parameters. That's mostly fine, until we get a WM_ERASEBKGND message with a wParam (a handle to a device context) which is > 0x7FFFFFFF, and the call to DefWindowProc fails at converting that to an int and passes the wrong value. So the window never renders certain items to the correct device context. My debug logging (which wraps Windows API functions to print their parameters to the console) didn't catch this either, since it was logging the value before it got mangled.
Changing the declaration to use uint seems to have resolved it. (And maybe I can use API Monitor's definition files for my bindings, since the ones I have seem to be somewhat inaccurate.)