Using GetOpenFile and GetKeyState in console application - winapi

Question

I am experiencing odd behavior when using GetKeyState (or GetAsyncKeyState, for that matter) in a console app. One aspect of the app is to ask the user for a file to open using GetFileOpen. At the end of the program, GetKeyState monitors the state of the spacebar. The GetKeyState (or GetAsyncKeyState) function never sets the high order bit whenever the spacebar is pressed. If I do not call GetOpenFile and then monitor GetKeyState, all works as expected.
Here are the two basic scenarios.
Scenario 1:
#include <windows.h>
int main(int argc, char *argv[])
{
char filename[ 512 ] = {0};
OPENFILENAME ofn = {0};
int filenameSize = 512;
char title[1000] = {0};
strcpy(title, "Open File");
ZeroMemory(&ofn, sizeof(OPENFILENAME));
ofn.lStructSize = sizeof(OPENFILENAME);
ofn.hwndOwner = null;
ofn.lpstrFile = filename;
ofn.nMaxFile = filenameSize;
ofn.lpstrFilter = "All files (*.*)\0*.*\0\0";
ofn.nFilterIndex = 1;
ofn.lpstrFileTitle = NULL;
ofn.nMaxFileTitle = 0;
ofn.lpstrInitialDir = NULL;
ofn.lpfnHook = NULL;
ofn.lpstrTitle = title;
ofn.Flags = OFN_FILEMUSTEXIST | OFN_HIDEREADONLY;
GetOpenFileName(&ofn); // filename obtained
WaitForSpaceBar(); // return value's upper bit is never set for
// GetKeyState(VK_SPACE);
return 0;
}
Scenario 2:
int main(int argc, char *argv[])
{
WaitForSpaceBar(); // returns immediately after spacebar is pressed
return 0;
}
WaitForSpaceBar code
void WaitForSpaceBar()
{
#define KEY_PRESSED_FLAG 1
SHORT spacePressed = GetKeyState(VK_SPACE);
printf("\nPress spacebar to continue...\n");
while (!(spacePressed & KEY_PRESSED_FLAG))
{
Sleep(1);
spacePressed = GetKeyState(VK_SPACE);
// for debugging purposes only
printf("spacePressed = 0x%04x\n", spacePressed);
}
}
The first scenario outputs "spacePressed = 0x0000" indefinitely regardless of how many times I press the spacebar.
The second scenario outputs "spacePressed = 0x0000" until the spacebar is actually pressed. Once pressed, the output is "spacePressed = 0xffffff81", and the program terminates.
Any ideas as to what is happening?

Answer 1

The GetKeyState function is useless if your are not pumping message.
From the documentation:
Remarks
The key status returned from this function changes as a thread reads
key messages from its message queue. The status does not reflect the
interrupt-level state associated with the hardware. Use the
GetAsyncKeyState function to retrieve that information.
Use the following code for your WaitForSpaceBar function (note the new value for KEY_PRESSED_FLAG):
void WaitForSpaceBar()
{
SHORT spacePressed = GetAsyncKeyState(VK_SPACE);
#define KEY_PRESSED_FLAG 0x8000
printf("\nPress spacebar to continue...\n");
while (!(spacePressed & KEY_PRESSED_FLAG))
{
Sleep(1);
spacePressed = GetAsyncKeyState(VK_SPACE);
// for debugging purposes only
printf("spacePressed = 0x%04x\n", spacePressed);
}
}

Answer 2

From the MSDN documentation, it looks like GetKeyState is only meaningful when your thread is pumping messages and calling it in response to a keyboard message.
The key status returned from this function changes as a thread reads key messages from its message queue.
I suspect that GetOpenFile spins up its own thread and that thread somehow becomes the main UI thread for your process (because it's the only one doing GUI work).
If you're using Windows API calls, you may as well use something console-specific like ReadConsole.
Update: I pasted the question code into VS 2010, and cannot reproduce the problem. The GetKeyState function is working as expected regardless of whether GetOpenFileName was called.