ToAscii/ToUnicode in a keyboard hook destroys dead keys - windows

Question

It seems that if you call ToAscii() or ToUnicode() while in a global WH_KEYBOARD_LL hook, and a dead-key is pressed, it will be 'destroyed'.
For example, say you've configured your input language in Windows as Spanish, and you want to type an accented letter á in a program. Normally, you'd press the single-quote key (the dead key), then the letter "a", and then on the screen an accented á would be displayed, as expected.
But this doesn't work if you call ToAscii() or ToUnicode() in a low-level keyboard hook function. It seems that the dead key is destroyed, and so no accented letter á shows up on screen. Removing a call to the above functions resolves the issue... but unfortunately, I need to be able to call those functions.
I Googled for a while, and while a lot of people seemed to have this issue, no good solution was provided.
Any help would be much appreciated!
EDIT: I'm calling ToAscii() to convert the virtual-key code and scan code received in my LowLevelKeyboardProc hook function into the resulting character that will be displayed on screen for the user.
I tried MapVirtualKey(kbHookData->vkCode, 2), but this isn't as "complete" a function as ToAscii(); for example, if you press Shift + 2, you'll get '2', not '#' (or whatever Shift + 2 will produce for the user's keyboard layout/language).
ToAscii() is perfect... until a dead-key is pressed.
EDIT2: Here's the hook function, with irrelevant info removed:
LRESULT CALLBACK keyboard_LL_hook_func(int code, WPARAM wParam, LPARAM lParam) {
LPKBDLLHOOKSTRUCT kbHookData = (LPKBDLLHOOKSTRUCT)lParam;
BYTE keyboard_state[256];
if (code < 0) {
return CallNextHookEx(keyHook, code, wParam, lParam);
}
WORD wCharacter = 0;
GetKeyboardState(&keyboard_state);
int ta = ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
keyboard_state, &wCharacter, 0);
/* If ta == -1, a dead-key was pressed. The dead-key will be "destroyed"
* and you'll no longer be able to create any accented characters. Remove
* the call to ToAscii() above, and you can then create accented characters. */
return CallNextHookEx(keyHook, code, wParam, lParam);
}

Answer 1

Quite an old thread. Unfortunately it didn't contain the answer I was looking for and none of the answers seemed to work properly. I finally solved the problem by checking the MSB of the MapVirtualKey function, before calling ToUnicode / ToAscii. Seems to be working like a charm:
if(!(MapVirtualKey(kbHookData->vkCode, MAPVK_VK_TO_CHAR)>>(sizeof(UINT)*8-1) & 1)) {
ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
keyboard_state, &wCharacter, 0);
}
Quoting MSDN on the return value of MapVirtualKey, if MAPVK_VK_TO_CHAR is used:
[...] Dead keys (diacritics) are indicated by setting the top bit of the return value. [...]

Answer 2

stop using ToAscii() and use ToUncode()
remember that ToUnicode may return you nothing on dead keys - this is why they are called dead keys.
Any key will have a scancode or a virtual key code but not necessary a character.
You shouldn't combine the buttons with characters - assuming that any key/button has a text representation (Unicode) is wrong.
So:
for input text use the characters reported by Windows
for checking button pressed (ex. games) use scancodes or virtual keys (probably virtual keys are better).
for keyboard shortcuts use virtual key codes.

Answer 3

Call 'ToAscii' function twice for a correct processing of dead-key, like in:
int ta = ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
keyboard_state, &wCharacter, 0);
int ta = ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
keyboard_state, &wCharacter, 0);
If (ta == -1)
...

Answer 4

Calling the ToAscii or ToUnicode twice is the answer.
I found this and converted it for Delphi, and it works!
cnt:=ToUnicode(VirtualKey, KeyStroke, KeyState, chars, 2, 0);
cnt:=ToUnicode(VirtualKey, KeyStroke, KeyState, chars, 2, 0); //yes call it twice

Answer 5

I encountered this issue while creating a key logger in C# and none of the above answers worked for me.
After a deep blog searching, I stumbled across this keyboard listener which handles dead keys perfectly.

Answer 6

Here is a full code which covers dead keys and shortcut keys using ALT + NUMPAD, basically a full implementation of a TextField input handling:
[DllImport("user32.dll")]
public static extern int ToUnicode(uint virtualKeyCode, uint scanCode, byte[] keyboardState, [Out, MarshalAs(UnmanagedType.LPWStr, SizeConst = 64)] StringBuilder receivingBuffer, int bufferSize, uint flags);
private StringBuilder _pressCharBuffer = new StringBuilder(256);
private byte[] _pressCharKeyboardState = new byte[256];
public bool PreFilterMessage(ref Message m)
{
var handled = false;
if (m.Msg == 0x0100 || m.Msg == 0x0102)
{
bool isShiftPressed = (ModifierKeys & Keys.Shift) != 0;
bool isControlPressed = (ModifierKeys & Keys.Control) != 0;
bool isAltPressed = (ModifierKeys & Keys.Alt) != 0;
bool isAltGrPressed = (ModifierKeys & Keys.RMenu) != 0;
for (int i = 0; i < 256; i++)
_pressCharKeyboardState[i] = 0;
if (isShiftPressed)
_pressCharKeyboardState[(int)Keys.ShiftKey] = 0xff;
if (isAltGrPressed)
{
_pressCharKeyboardState[(int)Keys.ControlKey] = 0xff;
_pressCharKeyboardState[(int)Keys.Menu] = 0xff;
}
if (Control.IsKeyLocked(Keys.CapsLock))
_pressCharKeyboardState[(int)Keys.CapsLock] = 0xff;
Char chr = (Char)0;
int ret = ToUnicode((uint)m.WParam.ToInt32(), 0, _pressCharKeyboardState, _pressCharBuffer, 256, 0);
if (ret == 0)
chr = Char.ConvertFromUtf32(m.WParam.ToInt32())[0];
if (ret == -1)
ToUnicode((uint)m.WParam.ToInt32(), 0, _pressCharKeyboardState, _pressCharBuffer, 256, 0);
else if (_pressCharBuffer.Length > 0)
chr = _pressCharBuffer[0];
if (m.Msg == 0x0102 && Char.IsWhiteSpace(chr))
chr = (Char)0;
if (ret >= 0 && chr > 0)
{
//DO YOUR STUFF using either "chr" as special key (UP, DOWN, etc..)
//either _pressCharBuffer.ToString()(can contain more than one character if dead key was pressed before)
//and don't forget to set the "handled" to true, so nobody else can use the message afterwards
}
}
return handled;
}

Answer 7

It is known that ToUnicode() and its older counterpart ToAscii() can change keyboard state of the current thread and thus mess with dead keys and ALT+NUMPAD keystrokes:
As ToUnicodeEx translates the virtual-key code, it also changes the
state of the kernel-mode keyboard buffer. This state-change affects
dead keys, ligatures, alt+numpad key entry, and so on. It might also
cause undesired side-effects if used in conjunction with
TranslateMessage (which also changes the state of the kernel-mode
keyboard buffer).
To avoid that you can do your ToUnicode() call in a separate thread (it will have a separate keyboard state) or use a special flag in wFlags param that is documented in ToUnicode() docs:
If bit 2 is set, keyboard state is not changed (Windows 10, version
1607 and newer)
Or you can prepare sc->char mapping table beforehand and update it on language change event.
I think it should work with ToAscii() too but better not use this old ANSI codepage-dependant method. Use ToUnicode() API instead that can even return ligatures and UTF-16 surrogate pairs - if keyboard layout have them. Some do.
See Asynchronous input vs synchronous input, a quick introduction
for the reason behind this.

Answer 8

I copy the vkCode in a queue and do the conversion from another thread
#HOOKPROC
def keyHookKFunc(code,wParam,lParam):
global gkeyQueue
gkeyQueue.append((code,wParam,kbd.vkCode))
return windll.user32.CallNextHookEx(0,code,wParam,lParam)
This has the advantage of not delaying key processing by the os

Answer 9

This works for me
byte[] keyState = new byte[256];
//Remove this if using
//GetKeyboardState(keyState);
//Add only the Keys you want
keysDown[(int)Keys.ShiftKey] = 0x80; // SHIFT down
keysDown[(int)Keys.Menu] = 0x80; // ALT down
keysDown[(int)Keys.ControlKey] = 0x80; // CONTROL down
//ToAscii should work fine
if (ToAscii(myKeyboardStruct.VirtualKeyCode, myKeyboardStruct.ScanCode, keyState, inBuffer, myKeyboardStruct.Flags) == 1)
{
//do something
}