Continually track mouse coordinates - ruby

I want to continually track the mouse as it moves, or at least every 0.5 seconds:
require "curses"
include Curses
init_screen
start_color
init_pair(COLOR_BLUE,COLOR_BLUE,COLOR_WHITE)
init_pair(COLOR_RED,COLOR_RED,COLOR_WHITE)
crmode
noecho
stdscr.keypad(true)
close = false
begin
mousemask(BUTTON1_CLICKED|BUTTON2_CLICKED|BUTTON3_CLICKED|BUTTON4_CLICKED)
count = 0
while( count < 10 )
sleep 0.5
m = getmouse
winx = Window.new(7,30,10,10)
winx.keypad = true
winx.box(?|, ?-, ?+)
winx.setpos(2,3)
winx.addstr "Loop Count: " + count.to_s
winx.setpos(3,3)
winx.addstr "Mouse Position: " + m.inspect
winx.refresh
count += 1
end
refresh
ensure
close_screen
end
I think I'm close, but for some reason, getmouse is returning nill? Why is this? Does getmouse only work after an event such as a click? If so, is it impossible to continually track the mouse?
This from the ruby documentation
getmouse() click to toggle source
Returns coordinates of the mouse.
This will read and pop the mouse event data off the queue
See the BUTTON*, ALL_MOUSE_EVENTS and REPORT_MOUSE_POSITION constants
REPORT_MOUSE_POSITION appears to be key here, but I really don't know how to use these constants. getmouse(REPORT_MOUSE_POSITION) doesn't work..sorry if that's majorly nooby, but there isn't much documentation out there.

You need to add REPORT_MOUSE_POSITION to your mousemask:
mousemask(BUTTON1_CLICKED|BUTTON2_CLICKED|BUTTON3_CLICKED|BUTTON4_CLICKED|REPORT_MOUSE_POSITION)
Or perhaps just
mousemask(ALL_MOUSE_EVENTS)
Curses processes mouse clicks in the same stream as key presses. So you need to get your mouse events with getch.
In your case, I recommend setting getch to non-blocking read stdscr.timeout=0 and adding a case statement:
case getch
when KEY_MOUSE
m = getmouse
winx.addstr "Mouse Position: #{m.x} #{m.y} #{m.z}"
end
Unfortunately on my system this only reports mouse movement when I click, so you might be out of luck depending on your curses implementation/terminal.
If you aren't too afraid of C, I recommend reading the ncurses C documentation. Ruby's curses library is basically a direct translation of it.

According to the TLDP nurses documentation, you need to use REPORT_MOUSE_POSITION Asa constant in your mouse mask. Try this:
mousemask(REPORT_MOUSE_POSITION|BUTTON1_CLICKED|BUTTON2_CLICKED|BUTTON3_CLICKED|BUTTON4_CLICKED)
This tells curses to report the mouse position as an event.

See the answer to Mouse movement events in NCurses.
Mouse event reporting depends heavily on the terminal emulator (on a Mac in iTerm2, I had to "Enable xterm mouse reporting" in preferences), and on the TERM settings as described above (I had to launch ruby like TERM=xterm-1003 ruby curses_mouse.rb)
Also, you must call getch and test for KEY_MOUSE before calling getmouse. For example, if you're only interested in mouse events:
c = getch
case c
when KEY_MOUSE
m = getmouse
end
Also, add |REPORT_MOUSE_POSITION to mousemask call.

Related

selenium scroll element into (center of) view

When an element is out of view with selenium and one tries to interact with it, selenium will usually scroll the element into view first implicitly. This is great except that what is annoying is that it usually puts in the element just enough into view. What I mean is that if the element is below the window, it will scroll down enough just till when the element is just bordering the edge of the window.
Usually this is fine, but when working on a web site with borders around it, this will lead to numerous of these kinds of errors
Selenium::WebDriver::Error::UnknownError:
unknown error: Element is not clickable at point (438, 747). Other element would receive the click: <body>...</body>
Because usually the border of the web page is over it, but will try to click the element anyway. Is there anyway handle this? perhaps to automatically move elements to the center of the screen when out of view? I am thinking along the lines monkey-patching via ruby.
This should work in order to scroll element into center of view:
WebElement element = driver.findElement(By.xxx("xxxx"));
String scrollElementIntoMiddle = "var viewPortHeight = Math.max(document.documentElement.clientHeight, window.innerHeight || 0);"
+ "var elementTop = arguments[0].getBoundingClientRect().top;"
+ "window.scrollBy(0, elementTop-(viewPortHeight/2));";
((JavascriptExecutor) driver).executeScript(scrollElementIntoMiddle, element);
Yes, it is possible to automatically scroll the browser such that any element we interact with gets centered in the window. I have a working example below, written and tested in ruby using selenium-webdriver-2.41.0 and Firefox 28.
Full disclosure: You might have to edit parts of your code slightly to get this to work properly. Explanations follow.
Selenium::WebDriver::Mouse.class_eval do
# Since automatic centering of elements can be time-expensive, we disable
# this behavior by default and allow it to be enabled as-needed.
self.class_variable_set(:##keep_elements_centered, false)
def self.keep_elements_centered=(enable)
self.class_variable_set(:##keep_elements_centered, enable)
end
def self.keep_elements_centered
self.class_variable_get(:##keep_elements_centered)
end
# Uses javascript to attempt to scroll the desired element as close to the
# center of the window as possible. Does nothing if the element is already
# more-or-less centered.
def scroll_to_center(element)
element_scrolled_center_x = element.location_once_scrolled_into_view.x + element.size.width / 2
element_scrolled_center_y = element.location_once_scrolled_into_view.y + element.size.height / 2
window_pos = #bridge.getWindowPosition
window_size = #bridge.getWindowSize
window_center_x = window_pos[:x] + window_size[:width] / 2
window_center_y = window_pos[:y] + window_size[:height] / 2
scroll_x = element_scrolled_center_x - window_center_x
scroll_y = element_scrolled_center_y - window_center_y
return if scroll_x.abs < window_size[:width] / 4 && scroll_y.abs < window_size[:height] / 4
#bridge.executeScript("window.scrollBy(#{scroll_x}, #{scroll_y})", "");
sleep(0.5)
end
# Create a new reference to the existing function so we can re-use it.
alias_method :base_move_to, :move_to
# After Selenium does its own mouse motion and scrolling, do ours.
def move_to(element, right_by = nil, down_by = nil)
base_move_to(element, right_by, down_by)
scroll_to_center(element) if self.class.keep_elements_centered
end
end
Recommended usage:
Enable automatic centering at the start of any code segments where elements are commonly off-screen, then disable it afterward.
NOTE: This code does not seem to work with chained actions. Example:
driver.action.move_to(element).click.perform
The scrolling fix doesn't seem to update the click position. In the above example, it would click on the element's pre-scroll position, generating a mis-click.
Why move_to?
I chose move_to because most mouse-based actions make use of it, and Selenium's existing "scroll into view" behavior occurs during this step. This particular patch shouldn't work for any mouse interactions that don't call move_to at some level, nor do I expect it to work with any keyboard interactions, but a similar approach should work, in theory, if you wrap the right functions.
Why sleep?
I'm not actually sure why a sleep command is needed after scrolling via executeScript. With my particular setup, I am able to remove the sleep command and it still works. Similar examples from other developers across the 'net include sleep commands with delays ranging from 0.1 to 3 seconds. As a wild guess, I would say this is being done for cross-compatibility reasons.
What if I don't want to monkey-patch?
The ideal solution would be, as you suggested, to change Selenium's "scroll into view" behavior, but I believe this behavior is controlled by code outside of the selenium-webdriver gem. I traced the code all the way to the Bridge before the trail went cold.
For the monkey-patch averse, the scroll_to_center method works fine as a standalone method with a few substitutions, where driver is your Selenium::WebDriver::Driver instance:
driver.manage.window.position instead of
#bridge.getWindowPosition
driver.manage.window.size instead of
#bridge.getWindowSize
driver.execute_script instead of
#bridge.executeScript
The following code will scroll until the element is in view,
WebElement element = driver.findElement(By.id("id_of_element"));
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", element);
Thread.sleep(500);
//do anything you want with the element
You could use an explicit scroll action via javascript here. In that case you would find the element (this part works already if I understand your question correctly), then scroll the window to a specified position, and then interact with the element.
In java that would be:
WebElement element = driver.findElement(By.id("tabs")).findElement(By.className("youarehere"));
Point p = element.getLocation();
((JavascriptExecutor) driver).executeScript("window.scroll(" + p.getX() + "," + (p.getY() + 200) + ");");

How do I allow the Leap Motion to control my cursor in an OS X application?

So I have a game for Mac OS X built in cocos2D.
I'm using gestures to simulate keyboard commands to control my character and it works really well.
I submitted my game to the AirSpace store and it got rejected on the grounds that the Leap should be used to control my menus as well which is fair enough I guess.
Thing is for the life of me I cannot figure out how this is done. There are no examples out there to show me how to implement it and nothing in the SDK example that makes it clear either.
Does anyone have any examples they'd care to share, I only need it to hijack my cursor and allow a click when held over a button. I really didn't think something so complex would be needed on simply for basic menu navigation.
If this is a Mac only game you should have access to the Quartz event api. This is the easiest way to generate mouse events in OS X...
I would recommend simply tracking the palm (hand) position, and moving the cursor based on that.
This is how I do my palm tracking:
float handX = ([[hand palmPosition] x]);
float handY = (-[[hand palmPosition] y] + 150);
The "+ 150" is the number of millimetres above the Leap device, to use as the '0' location. From there you can move the cursor based on the hand offset from 0.
The function I use to move the cursor (using Quartz):
- (void)mouseEventWithType:(CGEventType)type loc:(CGPoint)loc deltaX:(float)dX deltaY:(float)dY
{
CGEventRef theEvent = CGEventCreateMouseEvent(NULL, type, loc, kCGMouseButtonLeft);
CGEventSetIntegerValueField(theEvent, kCGMouseEventDeltaX, dX);
CGEventSetIntegerValueField(theEvent, kCGMouseEventDeltaY, dY);
CGEventPost(kCGHIDEventTap, theEvent);
CFRelease(theEvent);
}
and an example function call:
[self mouseEventWithType:kCGEventMouseMoved loc:CGPointMake(newLocX, newLocY) deltaX:dX deltaY:dY];
This call will move the mouse. Basically you just pass the new location of the mouse, and the corresponding deltas, relative to the last cursor position.
I can provide more examples such as examples for getting mouse location, clicking the mouse, or even a full mouse moving program...
EDIT 1:
To handle click and drag with Quartz, you can call the same function as above only pass in kCGEventLeftMouseDown.
The catch is that in order to drag you cannot call the kCGEventMouseMoved you must instead pass kCGEventLeftMouseDragged while the drag is happening.
Once the drag is done you must pass a kCGEventLeftMouseUp.
To do a single click (no drag) you simply call mouse down and then up right after, without any drag...

AutoIt ControlCommand is not working as expected

I'm using AutoIt to try and make a little hotkey application to work with Windows Journal so I can quickly select different colors.
It seems I'm very close and yet very far to getting the desired result. I've used the AutoIt tool to find the CommandID of the toolbar and the ID of the colors. Here is my code:
ControlCommand("[CLASS:JournalApp]","",113,"SendCommandID", 40178)
My problem is that the color will not be selected. It will be selected to the degree that the color will have the "selection" brackets around it, but the color that I draw with will still be the last color I've selected.
So I tried messing around and found that this code:
ControlCommand("[CLASS:JournalApp]","",113,"Check","")
It will indeed select the color, but it will only select the light blue color. I don't know why, but that is the color that is always being selected. I have not found a way to combine the selecting ability of "SendCommandID" with the checking ability of "Check"
Also, it is a ToolbarWin32 Control.
I figured it out myself.
Here's what I've learned:
ControlCommand("[CLASS:JournalApp]","",113,"Check","")
Has a serious weakness in that it there is no way to specify which button will be checked. At first it seemed to be random, but after a while of playing around I noticed that it did it at a specific coordinate relative to the client window. Why? I have no idea. But at least it's not random.
ControlCommand("[CLASS:JournalApp]","",113,"SendCommandID", 40178)
Has a weakness in that, while on the surface it appears to have successfully clicked the button. The button's function is not actually executed. For my specific circumstance, the color of the pen did not change after I used this, though it appeared to click on the button.
Here's my solution(s):
I looked around and found that AutoIt has a library specifically for dealing with ToolBarWin32 Classes. This is the library from GuiToolbar.au3. With this I found that I was able to do a few things. One, was that I could send clicks to the buttons and change the state of the buttons even. I found that changing the state of the buttons did nothing in relation to triggering an event and the clicking worked, but it had the weakness that it caused the mouse to flinch. This did not work because my pen was near my tablet as that has priority of mouse movement. So I had to raise my pen away from my tablet in order to use the hotkeys--not very convenient. Here was my code for that solution:
if WinActive("[CLASS:JournalApp]") Then
WinActivate("[CLASS:ToolbarWindow32; INSTANCE:2]", "")
$cmdId = "401"&$hotKeys[$key-1+$shift]
If $cmdId < 40172 or $cmdId > 40188 Then
Return
EndIf
$hWnd = ControlGetHandle("[CLASS:JournalApp]", "", 113)
_GUICtrlToolbar_ClickButton($hWnd, $cmdId)
EndIf
What I found after was that AutoIt's native ControlClick() was a lot more useful in that it didn't cause the mouse to flinch whatsoever. It triggered the mouseclick event directly. So that in combination with a nice command from the toolbar library made for a much cleaner solution. Here it is:
if WinActive("[CLASS:JournalApp]") Then
WinActivate("[CLASS:ToolbarWindow32; INSTANCE:2]", "")
$cmdId = "401"&$hotKeys[$key-1+$shift]
If $cmdId < 40172 or $cmdId > 40188 Then
Return
EndIf
ConsoleWrite($hotKeys[$key-1])
$hWnd = ControlGetHandle("[CLASS:JournalApp]", "", 113)
;get the coords of the button and control send a click
local $btnCoords= _GUICtrlToolbar_GetButtonRect($hWnd, $cmdId)
ControlClick("[CLASS:JournalApp]", "", "[CLASS:ToolbarWindow32; INSTANCE:2]","left",1,$btnCoords[0]+2,$btnCoords[1]+2)
EndIf

Cocoa: How to send proper mouse delta events?

I am making an application to control the mouse using an analog stick from a gamepad for Mac OS X (10.7.3).
Currently it is working pretty well, I can control the cursor in desktop and most games. But in Team Fortress 2, I cant control the aim, but can control the cursor in the menu. Mouse wheel and clicks works everywhere.
Another strange thing is that when I move the real mouse, the aim "jumps" the traveled distance from my "virtual mouse" before aiming normally, so it somewhat is receiving the events.
The game option "Raw mouse input" is disabled (I think it is not even supported in osx). And a similar application can controle the aim successfully.
I suspect the game is looking for "delta movement" or "relative movement" events or anything similar, but my code sets the position of the cursor using absolute positions. Not sute how to change this.
Here is the snippet of code used to send mouseMoved events:
EDIT: Crappy code removed!
.
EDIT:
Also, because I did this way, I need to check the screen bounds manually to prevent the cursor going Crazy. So in multi-screen setups, and when the user change the resolution, it gets worst. Would be so much better if I can just send the amount of movement and let the OS take care of constraining the cursor.
.
The question is:
I am doing the mouse move events the wrong way?
How can I fix this?
EDIT2:
So, that was just a stupid bug, sorry =P
I just forgot to call CGSetIntegerValueField(kCGEventMouseDeltaX, ...) (and Y) with the delta values... =P
Finally got this working, after just a few short hours of Googling and experimenting :) Looks like Rodrigo was trying to say that you have to call CGEventSetIntegerValueField on kCGMouseEventDeltaX and kCGMouseEventDeltaY on the mouse event after constructing it.
My code, using PyObjC:
from Quartz.CoreGraphics import (
CGEventCreate,
CGEventCreateMouseEvent,
CGEventGetLocation,
CGEventPost,
CGEventSetIntegerValueField,
kCGEventMouseMoved,
kCGHIDEventTap,
kCGMouseEventDeltaX,
kCGMouseEventDeltaY,
)
def mouse_delta(delta_x, delta_y):
"""
Send a MouseMoved event with the given deltaX and deltaY attributes,
but without changing the cursor location.
"""
# Might be a better way to get the current mouse location
x, y = CGEventGetLocation(CGEventCreate(None))
# The `mouseButton` parameter to CGEventCreateMouseEvent is ignored
# unless `mouseType` is kCGEventOtherMouse{Down,Dragged,Up},
# so let's just pass the value 0
event = CGEventCreateMouseEvent(None, kCGEventMouseMoved, (x, y), 0)
CGEventSetIntegerValueField(event, kCGMouseEventDeltaX, delta_x)
CGEventSetIntegerValueField(event, kCGMouseEventDeltaY, delta_y)
return CGEventPost(kCGHIDEventTap, event)

Detecting a single mouse click in MFC

In MFC a double-mouse click event triggers the following sequence of messages
WM_LBUTTONDOWN
WM_LBUTTONUP
WM_LBUTTONDBCLK
WM_LBUTTONUP
So responding to the WM_LBUTTONDBCLK message allows you to detect a double-click. But if I just want to detect a single-click how to I distinguish it?
But just looking at the WM_LBUTTONUP message isn't enough as it could be a single-click or it could be the first click of a double-click.
How can I successfully identify just a single-click?
(Please allow me to call these events Mouse Up and Mouse Down. My MFC is a little rusty. And there's this stuff called .NET who's been messing up my terminology lately ;-)
Short story: You don't simply want to know about Mouse Click. You need more.
Long story:
Although this is counter-intuitive, it appears that simply wanting a mouse-click is fairly uncommon. Most often, you'll want to perform some processing on Mouse Down and do some further processing on Mouse Up. The trick is that simply tracking Mouse Up messages is not enough: Mouse Down may not have happened in your window. Do you consider it a valid click then? Especially considering that the Mouse Down processing (such as selecting an item) did not occur.
Going further up the reasoning, you should not rely on receiving a Mouse Up after you processed Mouse Down: User may have moved the mouse and released the button somewhere else (think drag'n'drop), in which case, you don't receive the MouseUp event... unless you capture the mouse on MouseDown to make sure you get mouse event up to Mouse Up even if the mouse left your window.
All in all, you end up tracking Mouse Down, capture the mouse and when you receive Mouse Up, just check if you own the capture. If not, the mouse was either double-clicked (no 2nd mouse down) or Mouse Down happened somewhere else hence you most likely don't care about this Mouse Up.
In conclusion: There's no MouseClick message simply because you wouldn't go very far with it: You need to handle more messages and implement more mechanics anyway.
Oh! And if your dealing with an existing control which already handles all this items and selection stuff, such as a listview, chances are it provides with a similar custom notification such as Item Activate or Item Selection Changed.
I just tried this in Delphi, the behavior is the same: even when a double click is happening, a single click event is issued right after the first one of the two.
I solved it using a timer, which works like this:
deactivate timer on WM_LBUTTONDBLCLK (and set bDbl to true)
activate timer on WM_LBUTTONUP if bDbl==false
deactivate on WM_LBUTTONUP if bDbl==true (and reset bDbl)
I set the interval of the timer to the time returned by GetDoubleClickTime.
MSDN says:
The GetDoubleClickTime function
retrieves the current double-click
time for the mouse. A double-click is
a series of two clicks of the mouse
button, the second occurring within a
specified time after the first. The
double-click time is the maximum
number of milliseconds that may occur
between the first and second click of
a double-click.
If the timer happens to fire then you have the real click. In my case the double click interval is 500ms, so any "real click" will be delayed this long.
You could check WM_LBUTTONDOWN has not been called more than once before WM_LBUTTONUP. In practice Windows does this for you, in that if you get a WM_LBUTTONDBCLK you tend not to get a WM_LBUTTONUP.
You can use PreTranslateMessage() to count the messages as they appear. If you've received only the mouse messages corresponding to a single-click, and the system-configured time for double-clicking has expired, you can safely assume it's a single-click.
As far as I know there is no way to know that this is the case as it is happening, which makes sense -- until the time is expired, there's no way to know that a second click is or isn't coming.
that's a little tricky.
I would detect the WM_LBUTTONDOWN & WM_LBUTTONUP combo, store that event somewhere and set a timeout for a second or so. If there isn't a WM_LBUTTONDBCLK during that timeout then you have a single click.
This might imply you need to have another thread running but I think you could accomplish it with one thread.
I think the solution is to start a timer after the first click & then check the elapsed time after at the next immediate click, this will tell you if it is a single click or double click.
You typically look at #MLButtonUp and you would not have single click and double click behavior on the same mouse button.

Resources