We're making a user-space device driver for OS X that moves the cursor using Quartz Events, and we ran into a problem when games — especially ones that run in a windowed mode — can't properly capture the mouse pointer (= contain/keep it within the boundaries of their windows). For example, it would go outside the game window and click on the desktop or nearby inactive applications.
We could fix this if only we could detect when an active application calls CGAssociateMouseAndMouseCursorPosition.
How would you do this? Any ideas are appreciated.
I dont know if this can help you
There is an option called Focus Follows Mouse
Focus Follows Mouse - The Mouse pointer will grab automatically change focus to a new window inisde this one app if you mouse over it, instead of having to click a window to get focus, then clicking to do something.
http://wineskin.urgesoftware.com/tiki-index.php?page=Manual+4.6+Advanced+-+Options
I have written a few different mouse logical layers (for bridging different input devices, etc.). I have found that hooking into the OS level WM_INPUT event is a sure way of getting very real-time mouse position information. There is also a less rigorous solution of just polling the mouse data you need from one of Windows' very primitive DLLs. They are lightning fast. You could poll on a 10ms timer and never see performance loss on a modern machine.
Related
I will paint an example of what I am asking in crystal clear detail.
Say we are in a typical userland app, like painting software or text editing, and we decide to hover the mouse or click a combo box. From this input alone the mice's trackball will need to be fetched from the device driver on the low-level, but the video driver will also need to update the position of the pointer on the screen accordingly at the (seemingly) same time as follows:
As the picture illustrates, not moving the mouse, but calling the driver/updating the screen constantly works. However, what if we choose to both click the menu combo box/drop down list, and update the framebuffer feed constantly? Are both drivers called almost at the same time, and what calls them (the GUI desktop, the app directly, etc.). Because the mouse works ANYWHERE, on an app or not ... that makes me question if the mouse input is indefinitely system called, however, video output depends more on what the app is(like below, texpad).
The mouse driver has nothing to do with the video driver. You're getting the distinction wrong between what happens in the kernel space and user space.
The kernel has no idea of a concept of a 'mouse pointer'. It only exports an interface saying, "I have a input device that gives X and Y coordinates". Likewise with the video driver, it only exports an interface saying, "I have a device where you can output pictures" - it doesn't know what a mouse pointer is.
The GUI is all handled in user space and that's where you get the concept of drop down boxes, mouse pointers etc... Whatever program handles your GUI takes the kernel interfaces and does stuff with it, such as associating the input device with a mouse pointer and outputting video to the screen.
How the contents of the screen is actually pushed to the hardware will be driver dependent and doesn't involve user space. The GUI just tells the kernel, "I want the screen to look like this" and the kernel handles the rest.
Both drivers can certainly run in parallel if you have a multi-core CPU but, unless you're implementing the driver or scheduler, you don't need to know the details.
You can probably figure out why I am asking this question. Even if not, it's very simple.
My question is whether it is possible to detect the use of SetCursorPos() on one's own application, without scanning other running applications for any calls to this API.
For example, if I have my cursor in a window and I call SetCursorPos(), can this window in anyway know that the cursor placement is not directly from the mouse (raw input)?
I am not oblivious to the fact that you can 'know' whether a mouse input is raw simply by checking how the position alters; for example, if the position changes from 100(X) & 100(Y) to 500(X) and 500(Y), without moving through each individual location between these two, then with certainty, something has altered the mouse position.
If anyone of you know of a way to produce 'raw mouse input', without any application being able to tell the difference between the output from a function, and that from a mouse--if there is such a difference--then that'd suffice, too.
Of course, whenever I move my mouse, the operating system I am using detects this and then appropriately moves the cursor accordingly. In practice, I should be able to alter this low level functionality as to my will?
There is no way for a window to directly determine how the mouse was moved. External applications could be using SetCursorPos(), but they could also be using lower level functions like mouse_event() or SendInput() instead. By the time the notification reach the target window, the OS has already normalized the data and any source information is lost If you really needed to detect use of SetCursorPos() or other functions, you would have to directly hook into those functions in every running process. Alternatively, you might try registering for "Raw Input" via RegisterRawInputDevices() and see if you get a corresponding notification from the mouse hardware directy, assumine those simulating functions do not trigger Raw notifications as well.
How can I fire an automatic key press or mouse click event when a color appears on the screen
on other application or browser?
It depends a lot on what you want. Do you want to send the keys to
your Application
another fixed Application
Simulate a global keypress
Simulating keys globally
All of these will cause problems targeting a specific application and the active window changes.
SendKeys Sends Messages to the active app. It's a high level function taking a string which encodes a sequence of keys.
keybd_event is very low level and injects a global keypress. In most cases SendKeys is easier to use.
mouse_event simulates mouse input.
SendInput supersedes these functions. It's more flexible but a bit harder to use.
Sending to a specific window
When working with a fixed target window, sending it messages can work depending on how the window works. But since this doesn't update all states it might not always work. But you don't have a race condition with changing window focus, which is worth a lot.
WM_CHAR sends a character in the basic multilingual plane (16 bit)
WM_UNICHAR sends a character supporting the whole unicode range
WM_KEYDOWN and WM_KEYUP Sends keys which will be translated to characters by the keyboard layout.
My recommendation is when targeting a specific window/application try using messages first, and only if that fails try one of the lower level solutions.
when a color appears on the screen on other application or browser
I made one program using OpenCV and C++ for operating mouse with finger gesture. I used 3 color strips for 3 mouse function.
Yellow color for Left click
Blue color for Right click
Pink color for controlling cursor position
Whenever camera detect these colors, associated function takes place, I have used mouse_event for performing mouse function.
For more information you may read my code, blog, video.
I'm not 100% sure what you want, but if all you are after is running the method linked the the button.Clicked event, then you can manually run the method just like any other method.
You can use the .NET SendKeys class to send keystrokes.
Emulating mouse clicks requires P/Invoke.
I don't know how to detect colors on the screen.
I have been all over the web looking for an answer to this, and my question is this:
How does a GUI framework work? for instance how does Qt work, is there any books or wibsites on the topic of writing a GUI framework from scratch? and also does the framework have to call methods from the operating systems GUI framework?
-- Thank you to any one who takes the time to try to answer this question, and forgive me if i misspelled anything.
In the old days we did a lot of GUI programming from scratch. It is not as hard as it seems, but it requires a few weeks to come with results.
First you need a good drawing library. Minimal functionality for this library is drawing clipped rectangles (using patterns), lines, bitmaps, and fonts. You can cheat by creating fonts as bitmaps, and clipped rectangle is just a bunch of horizontal lines.
Now you need at least drivers for mouse, keyboard, and timer (if not already provided by the operating system). In general, you will need to detect keys, symbol keys (such as shift, etc.), mouse moves and mouse clicks. Basic timer functions will allow you to detect double clicks.
Then you need to create a window data structure. This data structure needs to have coordinates i.e. a rectangle, link to parent window (if not top window), and window function i.e. the function that will be called when this window should handle some events.
Once you can draw on screen you need some rectangle algebra functions. You need at least good function to calculate intersection of rectangles, and a quick resolution of relative to absolute coordinates. For example - if your child window has parent then its' x and y should recursively be added to parent x and y until you reach top window.
At this point you have your:
- primitive graphical functions,
- window structure,
- mouse driver, keyboard driver, and timer,
- rectangle arithmetic.
Now you can write your main event harvesting function. This function will run all the time. It's purpose will be to detect events and send messages to correct windows. What is an event? Well, when you start your program, store mouse x and y coordinates. Then in a loop check if they have changed. If they have changed, find the window at that position ... and send WM_MOUSEMOVE event to it. Your harvesting function should handle:
- mouse moves
- mouse clicks
- mouse double clicks (remember last click and position, measure time and decide if it is a double click or not)
- timer events
- keyboard buffer changes
...
Now you should be able to send events to windows. But you really need a mechanism for it. It is a combination of message queue, and window procedure. It usually works like this: each window has a window procedure which commonly accepts four arguments: message id (i.e. is it mouse move, is it paint message), window handle, parameter 1 and parameter 2. You can call this window procedure directly using something like a send_message functions. Or you can send this window a message via post_message function. This will put message to the queue and window will process messages one by one, eventually receiving this one. So why should you call one messages directly and put others to the queue? Because of priority. You see, a keyboard click can wait some time before being processed. But a window redraw must complete immediately to prevent flicker and wrong data on screen.
So your harvest_events function sends messages to windows using post_message, and send_message. And your window message pump gets them using typical message pump like this:
while (pmsg = get_message() != NULL) send_message(pmsg->id, pmsg->hwnd, pmsg->p1, pmsg->p2);
get_message simply obtains message from the queue, and calls send message. Simple, huh? Well, not quite so. This way you would only receive driver messages to windows, but you also need some functions to redraw windows, move them, etc.. When you create move_window function, resize_window, show_window, and hide_window function, your window coordinates will change. Parts of other windows will be uncovered (if top window is moved or closed).You need to calculate which windows are affected by coordinate changes and send paint message to those windows (to repaint only the parts that were uncovered - remember, you have clipping drawing functions so this will work).
These functions introduces messages msg_paint, msg_move, msg_resize, msg_hide...
Last, you need to maintain hierarchy of windows. Your top window should be the desktop. It should have child windows (application top windows). These windows may have further child windows (buttons, edit boxes, etc.) The obvious structure for holding these is the window tree. When you detect mouse click you have to traverse window tree and do it in a smart way (finding out who has focus, who is modal, etc.) to send message to the right window. And when you draw you also must traverse all children to see who is uncovered and who is not. Last but not least, you need to handle mouse rectangle as top window to prevent flickering the mouse as windows are re-drawn or (using timers and msg_paint events) animated.
That's roughly it.
A GUI framework like Qt generally works by taking the existing OS's primitive objects (windows, fonts, bitmaps, etc), wrapping them in more platform-neutral and less clunky classes/structures/handles, and giving you the functionality you'll need to manipulate them. Yes, that almost always involves using the OS's own functions, but it doesn't HAVE to -- if you're designing an API to draw an OpenGL UI, for example, most of the underlying OS's GUI stuff won't even work, and you'll be doing just about everything on your own.
Either way, it's not for the faint of heart. If you have to ask how a GUI framework works, you're not even close to ready to design one. You're better off sticking with an existing framework and extending it to do the spiffy stuff it doesn't do already.
Whenever I write mouse handling code, the onmousedown/onmouseup/onmousemove model always seemed to force me to produce unnecessarily complex code that would still end up causing all sorts of UI bugs.
The main problem which I see even in major pieces of software these days is the "ghost mouse" event where you drag to outside the window and then let go. Once you return back into the window, the application still thinks you have the mouse down even though the button is up. This is especially annoying when you're trying to highlight something that goes to the border of the screen.
Is there a RIGHT way to write mouse code or is the entire model just flawed?
Ordinarily one captures the mouse events on mouse down so the mouse move and mouse up go through your code regardless of the caret moving out of you application window.
More recently this is a problem when running a VM or remote session, its difficult for apps in these to track the mouse outside of the machine screen area represented by a window on a host.
I'm not sure what environment you're attempting to track mouse buttons in, but the best way to handle this is to have a mouse listener that tracks onmouseup 100% of the time after you've detected onmousedown.
That way, it doesn't matter what screen region the user releases the mouse button in. It will reset no matter where it happens.