Applescript - get the coordinates of an element - macos

I need to get the coordinates {X,Y} of an element in order to generate a click event on it (I can't do that by using "click" since System Events has problem doing that and returns: "missing value"). I'll use these coordinates to pass them to a command line tool that simulates a mouse click.
How can I achieve that?

Can you get the properties of the ui element? I think you can even if you can't actually click it. One of the properties should be "position" and another will be "size". Those 2 things should help you find the screen coordinates so you can click the element.
For example try this and look at the properties...
tell application "System Events"
tell process "Safari"
properties of UI element 1 of window 1
end tell
end tell
So if I wanted to know the coordinates of the middle of that element so you can click it this would give you the x and y coordinates of where to click...
tell application "System Events"
tell process "Safari"
tell UI element 1 of window 1
set p to position
set s to size
end tell
end tell
end tell
set xCoordinate to (item 1 of p) + (item 1 of s) / 2
set yCoordinate to (item 2 of p) + (item 2 of s) / 2
return {p, s, xCoordinate, yCoordinate}

Related

How to get index of UI?

I want to get index of UI. I did like following, but I couldn't. Is there any way?
source:
tell application "System Events"
tell process "System Preferences"
index of button 9 of scroll area 1 of window 1
end tell
end tell
result:
error
expected result:
9
The AS System Events UI Elements Suite does not contain a property like id or index for UI Elements. The call
button 9 of scroll area 1 of window 1
just targets the ninth button, not the button with id 9. You could also write
ninth button of scroll area 1 of window 1
The only way I can think of something returning the 9 would be to walk through all buttons with a repeat loop:
tell application "System Preferences" to activate
tell application "System Events"
tell process "System Preferences"
repeat with ix from 1 to count (buttons of scroll area 1 of window 1)
if name of button ix of scroll area 1 of window 1 = "Monitors" then
return ix
end if
end repeat
end tell
end tell
The result is 9 then.
Greetings, Michael / Hamburg

Applescript delay so long when run click button command

i was write a simple code to click share button on Notes applicaion
set upArrow to ASCII character 30
activate application "Notes"
tell application "System Events"
tell process "Notes"
set frontmost to true
click button 2 of group 2 of splitter group of window 1
keystroke upArrow
keystroke return
end tell
end tell
the problem is
click button 2 of group 2 of splitter group 1 of window 1 of application process "Notes" --> missing value
it need 6s to complete run this and return error missing value
but if i dont add keystroke or any command follow that, this code work perfectly
I also get that delay and it returns missing value. But could you click a menu bar item instead?
activate application "Notes"
tell application "System Events" to tell process "Notes"
click menu item "Email" of menu 1 of menu item "Share" of menu "File" of menu bar 1
end tell
set focused of button 2 of group 2 of splitter group 1 of window 1 to true
keystroke space
or
set value of attribute "AXFocused" of button 2 of group 2 of splitter group 1 of window 1 to true
keystroke space
Source:
https://lists.apple.com/archives/Accessibility-dev/2006/Oct/msg00013.html
There is a thread that seems to describe the same bug that appears to be limited to specific software. It is on MacScripter.net: Script delays on clicking button (controlling Sys Prefs pane)
Note the most recent post that describes a delay in FileMaker. It seems like the only work-around when software exhibits this bug is to do something else to open the new window. If the only way to run the desired function is via a button, one (terrible) option is clicking at coordinates relative to the corner of the front window. Obviously this will easily fail if the content can move relative to the corner or if a new version of the software moves the button desired. Any other ideas?
UPDATE, 2023-02-13:
In case anybody is still dealing with this bug, here's some useful AppleScript to get the coordinates of the object you'd like to click. You can then feed those into some coordinate-clicking utility, like cliclick (binary), Python code, Keyboard Maestro, and so on.
tell application "System Events"
set someObject to <REFERENCE TO YOUR OBJECT>
-- e.g. op's: button 2 of group 2 of splitter group of window 1 of application process "Notes"
set {xCoord, yCoord} to position of someObject
set {xSize, ySize} to size of someObject
end tell
set objOffset to round (minNum({xSize, ySize}) / 2) rounding down
set xClick to xCoord + objOffset
set yClick to yCoord + objOffset

AppleScript doesn't stop running after exit repeat

I have the following code. It runs, and i get my result, i think something wrong with the exit repeat, after it get the ipad name, the script is still keep running until time out.
Can anyone tell me what's wrong with the code? Thanks!
set deviceName to "iPad"
tell application "System Events"
tell process "iTunes"
activate
repeat with i from 1 to the count of (row of outline 1 of scroll area 2 of window "iTunes")
repeat with j from 1 to the count of static text of row i of outline 1 of scroll area 2 of window "iTunes"
set xxxx to the value of item j of static text of row i of outline 1 of scroll area 2 of window "iTunes"
if (xxxx contains deviceName) then
print xxxx
click row i of outline 1 of scroll area 2 of window "iTunes"
exit repeat
end if
--exit repeat
end repeat
end repeat
end tell
end tell
If you want to exit the script after the first iPad is found, replace exit repeat with return.
The problem (you ask for) is that you have an repeat in an repeat. That means when you're in the sub-loop and exit the repeat you will jump to the main loop. To exit from here you have to an exit repeat again.
Looking at your code I don't understand the nested repeat loop. You can remove the surrounded/main repeat and It will work as expected.
set deviceName to "iPad"
tell application "System Events"
tell process "iTunes"
activate
repeat with UIElement in rows of outline 1 of scroll area 2 of window "iTunes"
if (value of static text of UIElement as text) begins with deviceName then return select UIElement
end repeat
end tell
end tell
The reason I use an begins with is that contains will click the previous purshcase menu items.

Using AXIdentifier in UI Scripting for Applescript

10.7.4 OSX Lion
Applescript
I am working with an application (built in house and has no Applescript dictionary) that has a static text element I want to copy to the clipboard and send to another app but I'm having a hard time getting it to work.
The script I was using for targeting the element looked like this:
Tell application "System Events" to set frontmost of process "*application*" to true
Tell application "System Events"
Tell process "*application*"
Tell static text 1 of tab view 1 scroll area 1 of splitter group 1 of splitter group 1 of splitter group 1 of window 1
keystroke "a" using command down
delay 0.1
keystroke "c" using command down
delay 0.1
end tell
end tell
end tell
end tell
What would happen was that the wrong text from the wrong element was copied to the clipboard every time I clicked in a different spot on the application (there are numerous text fields).
I noticed in UI Accessor/Accessibility Accessor that each UI element in the application has a unique AXIdentifier value when you mouse over them.
Is there anyway to accomplishing what I am trying to do, using AXIdentifier values to target that element and copy the text from it?
Thanks for all the help this is my first post and I hope it was worthy! ~TheLarkInn
You can do this by using AppleScript's filtering. For example, to get the From: pop-up menu in a message composition window in Apple Mail, there is no accessibility description you can match on, however there is a unique AXIdentifier which you can match as follows:
tell application "System Events"
tell application process "Mail"
tell window 1
get first pop up button whose value of attribute "AXIdentifier" is "popup_from"
end tell
end tell
end tell
This is more efficient than looping in AppleScript as it only involves sending one Apple Event to System Events.
I don't think there is a way to directly do what you're trying to do. It seems like you can only access attributes once you have a handle on an element via selector. Here is a very ugly solution that does what you're asking by iterating over all UI elements, but it is really slow with bigger UIs and probably not ideal for any production level code.
tell application "System Events"
tell process "Some Process"
set tElements to entire contents of window "Some Window"
repeat with tElement in tElements
if (exists attribute "AXIdentifier" of tElement) then
if value of attribute "AXIdentifier" of tElement = "Some AXIdentifier" then set tText to value of tElement
end if
end repeat
end tell
end tell
tText
I think using UIElementInspector or Accessibility Inspector from Xcode to build a selector string is the way to go!
Tell application "*application*" to activate
Tell application "System Events"
Tell application process "*application*"
set textStaticTextValue to value of static text 1 of tab view 1 scroll area 1 of splitter group 1 of splitter group 1 of splitter group 1 of window 1
end tell
end tell

Positioning a window with AppleScript using dual monitors

I have two monitors set up and I am trying to position the window of an application in the second monitor but nothing I do seems to work. For example I am using my laptop and the terminal window is maximized on the screen. Then I plug in an external monitor. I then want to run the applescript and have the terminal maximize on the larger second monitor.
Here is what I have right now:
set monitorTwoPos to {1050, -600}
set monitorTwoSze to {1200, 1920}
tell application "Microsoft Outlook"
set position of window 1 to monitorTwoPos
set size of window 1 to monitorTwoSze
end tell
Here is the error I get:
/Users/vcutten/AppleScripts/SpacesWork.scpt:1291:1332: execution error:
Microsoft Outlook got an error: Can’t make position of window 1 into type specifier. (-1700)
I'm pretty sure I'm just using set position and set size completely wrong :( When I used bounds it kind of works...
Bonus Question:
How can I loop through the open windows and get their size? Thanks!
What have you tried?
I think to solve this you need to calculate the screen size and coordinates of the second monitor. For example, your main monitor starts at position {0,0}. So the starting position of the second monitor has to be something different and you need to find that. Luckily I have written a tool that will give you both the starting coordinates and screen size of your monitors. Once you have the size and position then it's simple. System events can set the size and position of a window so you could do something like this...
set monitorSize to {800, 600}
set monitorPosition to {-800, 0}
tell application "System Events"
tell process "Terminal"
set frontWindow to first window
set position of frontWindow to monitorPosition
set size of frontWindow to monitorSize
end tell
end tell
So from the above script you just need the size and position variables. You can get my tool here called hmscreens which will give you those. You may need to do some adjusting of the coordinates depending on if the screen is measured from the lower left corner or upper left, but that's just simple math.
I hope that helps...
Use bounds instead of position, it works. You can get bounds of the window like this:
tell application "Microsoft Outlook"
get bounds of first window
end tell
Answer to the bonus question:
tell application "Microsoft Outlook"
repeat with nextWindow in (get every window)
get bounds of nextWindow
end repeat
end tell
If you open Replies tab at bottom part of Applescript editor, you will see all get results.
Hope it helps.
Here is a script that handles saving and restoring size and postion for multiple display configurations. It may have some issues with fullscreen apps but it seems to work ok.
-- allSettings is a list of records containing {width:? height:? apps:{{name:? pos:? size:?},...}
-- for each display setup store the apps and their associated position and size
property allSettings : {}
-- create a variable for the current settings
set currentSettings to {}
display dialog "Restore or save window settings?" buttons {"Restore", "Save"} default button "Restore"
set dialogResult to result
tell application "Finder"
-- use the desktop bounds to determine display config
set desktopBounds to bounds of window of desktop
set desktopWidth to item 3 of desktopBounds
set desktopHeight to item 4 of desktopBounds
set desktopResolution to desktopWidth & "x" & desktopHeight
-- find the saved settings for the display config
repeat with i from 1 to (count of allSettings)
if (w of item i of allSettings is desktopWidth) and (h of item i of allSettings is desktopHeight) then
set currentSettings to item i of allSettings
end if
end repeat
if (count of currentSettings) is 0 then
-- add the current display settings to the stored settings
set currentSettings to {w:desktopWidth, h:desktopHeight, apps:{}}
set end of allSettings to currentSettings
--say "creating new config for " & desktopResolution
else
--say "found config for " & desktopResolution
end if
end tell
tell application "System Events"
if (button returned of dialogResult is "Save") then
say "saving"
repeat with p in every process
if background only of p is false then
tell application "System Events" to tell application process (name of p as string)
set appName to name of p
if (count of windows) > 0 then
set appSize to size of window 1
set appPosition to position of window 1
else
set appSize to 0
set appPosition to 0
end if
set appSettings to {}
repeat with i from 1 to (count of apps of currentSettings)
if name of item i of apps of currentSettings is name of p then
set appSettings to item i of apps of currentSettings
end if
end repeat
if (count of appSettings) is 0 then
set appSettings to {name:appName, position:appPosition, size:appSize}
set end of apps of currentSettings to appSettings
else
set position of appSettings to appPosition
set size of appSettings to appSize
end if
end tell
end if
end repeat
end if
if (button returned of dialogResult is "Restore") then
if (count of apps of currentSettings) is 0 then
say "no window settings were found"
else
say "restoring"
repeat with i from 1 to (count of apps of currentSettings)
set appSettings to item i of apps of currentSettings
set appName to (name of appSettings as string)
try
tell application "System Events" to tell application process appName
if (count of windows) > 0 then
set position of window 1 to position of appSettings
set size of window 1 to size of appSettings
end if
end tell
end try
end repeat
end if
end if
end tell
https://gist.github.com/cmackay/5863257

Resources