Click on a specific place on a window, click on a specific button and control if the window changes - window

Actually I have 3 questions about the same problem: controlling a window with applescript.
What should I do if I would press on button "Close Window" of application "Google Chrome"?
Is it possible to check if the window changes? For example, to see if appear a pop-up or something like that...
What about clicking on a specific place into a window? I mean, I know I can use
tell application "System Events"
click at {x,y}
end tell
but this command use the entire screen as reference system, and I want it works only on a specific window. For example, if at "{x,y}" i put "{1,1}", applescript will click on the first item on the menu bar. Is there a way I can say to "System Events" to click at "{1,1}", but on the window "Google Chrome"?

Here are three examples of how to close the front window of Google Chrome using AppleScript:
Note: The following assumes Google Chrome is running with at least one window open when you test each example AppleScript code in Script Editor.
Example one is the most straight forward way:
tell application "Google Chrome" to close front window
Example two directly clicks the close button:
tell application "System Events" to tell ¬
application process "Google Chrome" to ¬
click button 1 of front window
Example three calculates the center of the close button and clicks there:
activate application "Google Chrome"
delay 0.5
tell application "System Events" to tell ¬
application process "Google Chrome" to tell ¬
front window
set posB1 to (position of button 1)
set szB1 to (size of button 1)
set x to (item 1 of posB1) + (item 1 of szB1) / 2 as integer
set y to (item 2 of posB1) + (item 2 of szB1) / 2 as integer
end tell
tell application "System Events" to click at {x, y}
Note that in the first two examples, the front window of Google Chrome doesn't even need to be the frontmost window on the Desktop; however, with the third example it does, otherwise the click at {x, y} will not go to the intended target.
That said, example three really shouldn't be used when there it a straight forward way, as in example one, to get the job done. Example three was just a proof of concept to get the coordinates to click at. This method may be useful in some fringe cases, especially in an app that doesn't directly support AppleScript.
Note: The example AppleScript code is just that and does not contain any error handling as may be appropriate. The onus is upon the user to add any error handling as may be appropriate, needed or wanted. Have a look at the try statement and error statement in the AppleScript Language Guide. See also, Working with Errors.

In applescript GUI scripting you can simply refer to an element by name or index and tell it to click or to perform an action. For instance to click the close button on the first open window in Chrome you could use:
tell application "System Events"
tell process "Google Chrome"
tell window 1
tell button 1
click
end tell
end tell
end tell
end tell
You don't actually need to know its physical position to click one it; you just need to know that the first button in the window is the close button.
System Events always returns the position of any element in screen pixels, so if you want the position of an element in terms of its window, get the position of the element, get the position of the window, and do some addition or subtraction (e.g., if you want to click at {5,5} in a window whose position is {100, 125}, click at {105, 130})
AppleScript isn't really designed to monitor GUI changes, though if you want to be tricky and you know what change you're looking for you can do something like this:
tell application "System Events"
tell process "..."
tell window 1's pop up button 3
repeat until (exists menu 1)
delay 0.2
end repeat
-- menu 1 now exists, so the pop up button is open
end tell
end tell
end tell
...but note that this will hang the script until the menu is opened. A more elegant way to handle that is to write a script application with an idle handler, like so:
on run
-- whatever initialization is needed
end run
on idle
tell application "System Events"
try
tell process "..."
tell window 1's pop up button 3
if exists menu 1 then
-- menu 1 now exists
-- the pop up button is open
-- do what must be done
end if
end tell
end tell
on error errstr
display alert "Something went wrong" message "The script sent this error: " & errstr
end try
end tell
return 0.2
end idle
You can leave that running in the background watching for specific changes in the GUI (the 'try' statement is in case the app you're watching quits, the window closes, or something unexpected happens to the GUI).
If you haven't already, open the System Events scripting definition in Script Editor and look at the Processes Suite. That will show you all the things you can do with GUI scripting.

Related

How to take a portion screenshot by using QuickTime Player and save it to the clipboard in macOS when using Apple Script for automation?

I'm a uni student taking online courses, I take portion screenshots by pressing Command+Control+Shift+4 every time to take notes, then I paste it to Notion. But it's not efficient to select the same area every time, then I found that QuickTime Player will keep the same area for capturing a portion screenshot, so I decided to use Apple Script to complete the operation.
The code I wrote could switch to the application QuickTime Player and then proceed to "New Screen Recording" in the menu by pressing Command+Control+N, but the problem is that sometimes pressing twice space will call out "Capture Selected Window" instead of "Capture Selected Portion". Meanwhile, the location where saving the screenshots will also be changed too though I already ticked "Remember Last Selection", maybe resetting the location to be saved every time is essential.
on run {input, parameters}
activate application "QuickTime Player"
tell application "System Events"
key code 45 using {command down, control down}
delay 0.1
# Switch to "Capture Select Portion" by pressing SPACE
repeat 2 times
key code 49
delay 0.1
end repeat
# Press ENTER to capture a screenshot
key code 35
delay 0.1
end tell
return input
end run
I spent a few hours finding the existed articles but perhaps not that fulfill my needs.
do shell script "screencapture -ci" is as same as pressing Command+Control+Shift+4, and I have to select the portion manually.
do shell script "screencapture -x -R20,20,640,380 ~/Desktop/test.png" needs a precise coordinate. But the coordinate is variable since the window is not fixed, and it needs to be found by using external software, maybe it's a little bit not efficient. If there is no solution for that, I will use it since it's much more convenient than selecting the portion every time.
Sorry about pasting others' code here and saying why they r not suitable for me, I just wanted to make it clear... Thanks for reading here, have a nice day no matter if u r willing to help me ^ ^
The part about pasting screenshots to Notion works, so I didn't paste it here.
I tried writing something, it works but still has some bugs.
For example, the coordinate of the menu "Options" should be initialized depending on the precise location of the bar since I didn't succeed in finding the element by using entire contents.
You can't move your mouse during the use, or it might cause an error:
System Events got an error: Can’t get menu \"Options\" of button \"Options\" of window \"Window\" of application process \"screencaptureui\"." number -1728 from menu "Options" of button "Options" of window "Window" of application process "screencaptureui
When selecting the path where screenshots will be saved, it might take a few seconds to let the program find the menu item Clipboard.
tell application "System Events"
tell process "QuickTime Player"
# Switch to QuickTime Player (already opened expected, no launching time reserved)
activate
# Open "New Screen Recording" from the menu
click menu item "New Screen Recording" of menu "File" of menu bar item "File" of menu bar 1 of application process "QuickTime Player" of application "System Events"
delay 1
end tell
# Capture Selected Portion
tell application process "screencaptureui"
click checkbox "Capture Selected Portion" of window "Window" of application process "screencaptureui" of application "System Events"
# Select Options from the menu (idk why if I replace it with click menu, the clipboard won't be found, needs to be fixed)
click at {817, 824}
delay 0.1
# Save to the Clipboard
click menu item "Clipboard" of menu "Options" of button "Options" of window "Window" of application process "screencaptureui" of application "System Events"
delay 1
end tell
# Enter to capture the screenshot
key code 36
delay 0.1
# Switch to Notion (already opened expected, no launching time reserved)
tell application "Notion"
activate
delay 0.1
end tell
# Paste the screenshot to Notion
key code 9 using {command down}
end tell
Hope this helps someone, if u hv any idea plz feel free to modify it since it still has bugs.

Applescript - Idein

I'm newbie to Appplescript. I need to automate certain actions on my computer related with my Bluetooth keyboards.
I want to be able to click on the remove or connect button of a keyboard in the following dialog window of the System Preferences Panel.
Dialog window
My code until this moment is as follows:
try
tell application "System Preferences"
activate
set current pane to pane "com.apple.preference.keyboard"
end tell
tell application "System Events"
tell process "System Preferences"
click button "Set Up Bluetooth Keyboard…" of window "Keyboard"
end tell
end tell
tell application "System Events"
tell group 1 of window 1 of application process "System Preferences"
click button "remove" of "Home Keyboard"
end tell
end tell
end try
My problem is related with the remove button since is unidentified cell of an unidentified table. With unidentified, I mean without description. Maybe there is an easy solution, but I'm not able to find it. Furthermore, It could happen that more than one keyboard exists, so I need to identify the cell from the Keyboard name.
Do you know any hint related with this issue?
Thanks in advance
Here is a sample script I used to reconnect a specific mouse via Bluetooth :
tell application "System Events"
tell application "System Preferences"
activate
reveal anchor "MouseTab" of pane id "com.apple.preference.mouse"
end tell
tell application process "System Preferences"
click button "Configuration of Bluetooth mouse…" of window 1 -- see note 1
delay 1
select (first row of table 1 of scroll area 1 of sheet 1 of front window whose value of item 1 of static text of UI element 1 contains "Mouse") -- see note 1
get value of item 1 of static text of UI element 1 of row 2 of table 1 of scroll area 1 of sheet 1 of front window
click button "Done" of sheet 1 of front window -- see note 1
end tell
tell application "System Preferences" to quit
end tell
Note 1 : Be careful about the 3 lines with comment 'see note 1' : the value of the string may be different for your local language. Please adjust these 3 values.
I think for keyboard, concept should be very similar. Because it is using GUI scripting, if Apple changes the layout of Bluetooth screen preferences, it must be adjusted. This script works from Yosemite to ElCaptain.I can't test it for next systems.

How can I click OK with Apple Script?

I'm trying to select OK with the popup for scaling the screen with Apple script.
Does anyone know what I can add to this script to allow me to click OK?
Apple Script:
tell application "System Preferences"
reveal anchor "displaysDisplayTab" of pane "com.apple.preference.displays"
end tell
tell application "System Events" to tell process "System Preferences" to tell window "Built-in Retina Display"
click radio button "Scaled" of radio group 1 of tab group 1
click radio button 1 of radio group 1 of group 1 of tab group 1
(* Trying to click the ok button here *)
delay 1
click ((pop up buttons of sheet 1 of window 1) whose description is "OK")
end tell
quit application "System Preferences"
Or also rather does anyone know a trusted application that can graphically print out the labels of the UI when I visit them on my screen so I will know what to use when I say: "tell UIName"?
Thanks.
Solution 1: Press Return
Since you're using System Events anyway, and the button in question is selected by default (indicated by being highlighted in blue), just make sure that the preference pane is in focus and use System Events to hit the return key:
tell application "System Events" to keystroke return
It's quick, simple, and saves the hassle of identifying UI elements within the hierarchy. The drawback is that the preference pane has tone in focus, and if it loses focus before it receives the keystroke, the rest of the script will fail.
Solution 2: The One You Asked For
Regarding identification of UI elements within the GUI objects hierarchy, you got the class of the button in question wrong as well as its description.
The "OK" button is referenced like so:
tell application "System Events" to tell process "System Preferences" ¬
to get button "OK" of sheet 1 of window "Built-in Retina Display"
(window 1 is also fine). You can use the whose filter to target it instead, which would be done like this (via System Events and the System Preferences process):
get buttons of sheet 1 of window 1 whose name is "OK"
but all this does is ask AppleScript to search for a button whose name we know, then annoyingly give the result back to us as a list (the list structure can be flattened by requesting the first button of sheet 1...).
However, we do know its name and that there's only one of it, so we can reference it by name directly.
As a side-bar, if you quickly run this command:
get the properties of button "OK" of sheet 1 of window "Built-in Retina Display"
you'll be able to see that its description is merely "button", which isn't what you had hoped. Now run this command:
get the actions of button "OK" of sheet 1 of window "Built-in Retina Display"
which reveals it has one action available, AXPress (this is equivalent to a mouse click).
Therefore, finally, the way to click the button in a more satisfactory way than hitting the return key looks like this:
tell application "System Events" to tell process "System Preferences" ¬
to tell button "OK" of sheet 1 of window "Built-in Retina Display" ¬
to perform action "AXPress"
For exploring the GUI objects on the screen, I occasionally use the Accessibility Inspector that comes with Apple's XCode. It's vaguely useful, although inconveniently annoying with some of the discrepancies between the names by which it references objects and the names by which AppleScript does (they're also really subtle discrepancies, but enough to stop your script from working and send you into a frenzy for weeks).
So, actually, I just explore it programmatically myself in Script Editor.
What I did to solve your issue was to bring up the pane and the dialog box in question, then systematically tell the System Preferences process to get UI elements; then get buttons of UI elements; then get buttons of UI elements of UI elements, and stop when I saw a pair of buttons returned called "OK" and "Cancel". It can be pain-staking, but it'll get you the correct reference.
There are other ways too, but I would be moving beyond the scope of this question.

Speed up AppleScript UI scripting?

I'm using NetShade as a proxy service and thought I could try to automate the switching between the different proxies as a nice start for my first AppleScript script.
The NetShade-app has no AppleScript support, so I have to use UI scripting. After a few tries (and some posts here) I managed to have a script, that switches the proxies via the menu bar item (here is a picture of it, since I can't post it inline due to reputation limit).
Unfortunately my code is extremely slow (≈6sec), which makes it kind of impractical as a script. The first menu opens immediately, but the selection of the sub-menu and the proxy server takes several seconds.
I'm using the following code:
set theProxy to "Netshade US 4"
tell application "System Events" to tell process "NetShade"
tell menu bar item 1 of menu bar 2
click
tell menu item "NetShade Proxy" of menu 1
click
tell menu item theProxy of menu 1
click
end tell
end tell
end tell
end tell
I already tried to add ignoring application responses, like suggested in a different thread (link), but that didn't help.
So finally my questions:
Is there a way to speed the process up? Maybe even a way to do all this in the background, without showing the menu items?
P.S.: I'm running OS X 10.9.1
Summary of the fix
To remove delay you need to do two things:
(I) Identify the click which is causing the delay and enclose only that line in the ignoring application responses block as shown below. In my case, it was click bt after which the execution was going into a wait mode for 5 to 6 seconds.
ignoring application responses
click bt
end ignoring
(II) I then also had to kill System Events to and start it again using the following commands.
do shell script "killall System\\ Events"
delay 0.1
-- Rest of the code to click stuff or send keycodes
This resolved the delay issue.
Details
I was having the same problem where I created a script to connect/disconnect my bluetooth headset through AppleScript. The script is given below.
tell application "System Events" to tell process "SystemUIServer"
set bt to (first menu bar item whose description is "bluetooth") of menu bar 1
click bt
tell (first menu item whose title is "SBH80") of menu of bt
click
tell menu 1
if exists menu item "Disconnect" then
click menu item "Disconnect"
else
click menu item "Connect"
end if
end tell
end tell
end tell
The script was working fine but had a problem where it would wait for 5 to 6 seconds after executing "click bt" above. I modified the code as follows and it is working absolutely fine now without any delay.
tell application "System Events" to tell process "SystemUIServer"
set bt to (first menu bar item whose description is "bluetooth") of menu bar 1
ignoring application responses
click bt
end ignoring
end tell
do shell script "killall System\\ Events"
delay 0.1
tell application "System Events" to tell process "SystemUIServer"
tell (first menu item whose title is "SBH80") of menu of bt
click
tell menu 1
if exists menu item "Disconnect" then
click menu item "Disconnect"
else
click menu item "Connect"
end if
end tell
end tell
end tell

How to select speakers for iTunes from applescript

I would like to be able to select output sound device for iTunes from a script (any programming language would be ok in fact).
For the moment I was able to use UI element scripting to get up to clicking on the button which gives the menu to select the speakers:
tell application "System Events"
tell window "iTunes" of process "iTunes"
set chbtn to first UI element whose help is "Choose which speakers to use."
tell chbtn
click
-- tell menu 1 to get every menu item
end tell
end tell
end tell
This works, and menu with possible choices appears. However, the applescript seems to stop after the click command, and further actions (in the place where the comment is in the code) happen only after I click somewhere on the screen myself. How can I prevent this and continue to select the menu item from this menu?
Any solution without reverting to UI scripting is also very welcome!
The solution code is
tell application "iTunes" to activate
tell application "System Events"
tell window "iTunes" of process "iTunes"
click (first UI element whose help is "Choose which speakers to use.")
keystroke "DENON" & return -- Select "DENON" airplay entry
-- keystroke "Computer" & return -- Select standard output
end tell
end tell
However, there is an annoying 4 second delay between the click and the keystroke
I've had that delay problem before when using UI scripting. You might be able to eliminate it by telling the script which elements to click. I don't have external speakers, so the elements' names and properties aren't on my computer. An easy way to get more info on the elements available is to use (not free, but excellent) UI Browser http://pfiddlesoft.com/uibrowser/. A less easy, but free, way to get more info on the elements is to use:
tell application "System Events"
tell window "iTunes" of process "iTunes"
set chbtn to first UI element whose help is "Show or hide item artwork and video viewer."
tell chbtn
entire contents
end tell
end tell
end tell

Resources