I wrote an AppleScript to synch my Reminders (via export to JSON). It runs great... from the Script Editor. As soon as I tried to run it on the command line via osascript, I discovered it hits a wall when it tries to access reminders. After maybe a minute and a half, I get this error:
/Users/robleach/Temporary/synchRemindersTest.scpt: execution error: Reminders got an error: AppleEvent timed out. (-1712)
I also noted these errors in the console:
error 19:33:49.628309-0400 tccd Refusing client without path (from responsibility_get_responsible_audit_token_for_audit_token) PID[1422]: (#3) No such process
error 19:33:49.628370-0400 tccd Refusing TCCAccessRequest for service kTCCServiceReminders from invalid client with pid 1422
Assuming it was a permissions issue, I looked in System Preferences>Security & Privacy>Reminders, and noted that osascript is not there, nor do I have ± buttons to add it, even after authenticating.
I wondered whether saving the script as an app would prompt the Security stuff to prompt me to enable it - and that I could cron the opening of that app, but if I do that, I won't be able to pass arguments to the script (or at least, I don't know how to do that). Plus, I'd rather everything happen in the background, without a dock icon or anything (aside from the Reminders app needing to be open).
I wrote a toy example of the script that produces the timeout error:
TOY 1
tell application "Reminders"
return (properties of every reminder whose completed is false)
end tell
And I call it like this:
> osascript /Users/robleach/Temporary/synchRemindersTest.scpt
Is there a way to allow the osascript run of the script to be allowed access to the reminders? Could I like code-sign the command line executable or something? If I were to write this in another language, would it have the same issue?
I'm running Catalina 10.15.7.
UPDATE 1
I dug some more into the console. There are number of other, potentially relevant errors. I think it actually is a timeout. When I run it in Script Editor, it runs in about 40 seconds, but it times out when I run it via osascript (at maybe around a minute and a half).
However, I recalled that I have another script on a cron job that accesses reminders, and I did not remember it having an issue. So I tested it, and for whatever reason, it executes a very similar command, but succeeds. And it runs much faster in the Editor than it does on the command line from osascript. I pulled that one line out of that script that succeeds and wrapped it in a toy script:
TOY 2
tell application "Reminders"
set theList to "ToDo Home Recurring"
set namesDates to {name, due date} of (every reminder in list theList whose completed is false)
display dialog "Got " & (count of namesDates) & " reminder names & dates"
end tell
and it failed via osascript with the same timeout error. Then I pulled out another line and added it to the toy:
TOY 3
tell application "Reminders"
set theList to "ToDo Home Recurring"
show list theList
delay 0.25
set namesDates to {name, due date} of (every reminder in list theList whose completed is false)
tell application "System Events" to display dialog "Got " & (count of namesDates) & " reminder names & dates"
end tell
...it succeeds. So I don't think this is a permissions issue any longer. This feels like it maybe has to do with how efficiently osascript accesses Reminders.
I also started noticing that my first toy example above sometimes fails when run from the script editor. I kept retrying it to get the running time I pasted above and I started to sense a pattern. I think that when I select a new list to view in the Reminders GUI (it doesn't matter which one), then run the script (from the editor), it works. But if I don't select a new list to view and run the script from the editor again, it fails with that timeout.
...but that seems nuts. Can anyone explain what's going on here?
NOTE: The script I'm writing is actually an AppleScript re-write of a Siri Shortcut I wrote (that runs reliably in around 25ish seconds). Since I wanted to automate it and run it more than once a day, I decided to use AppleScript.
UPDATE 2
I tried the answer suggested by #Robert Kniazidis.
TOY 4 (modification of TOY 1)
with timeout of 3600 seconds
tell application "Reminders"
set allRems to (properties of every reminder whose completed is false)
display dialog "Got " & (count of allRems) & " reminders"
end tell
end timeout
...and kept a closer watch on the console.
Attempt 1 (TOY 4)
I ran TOY 4, starting at 7:25:24, for 10 minutes, then control-c'd it. I immediately saw numerous errors in the console. I searched the console for "Reminders" and here's what I go during the run.
Attempt 2 (TOY 4)
Then, given my insights regarding my annecdotal successes when I click a list name in the Reminders GUI, I tried clicking on a random list and ran TOY 4 again immediately after. I started TOY 4 at 7:38:23. At 7:44:22, it succeeded! That's about 6 minutes!
The console had many fewer messages in it, none of them marked as error. For comparison's sake, here are the console results from searching for "Reminders".
Discussion
I have revised my theory about what's happening. Given the console messages, I infer that when you run by osascript from the command line, the script is identified as "indirect access", and as such, is subject to a higher level of security-scrutiny, thus is takes much longer to execute. Perhaps when I "click in the GUI" (or even via AppleScript, show list theList), the security concern is still considered "indirect", but is not totally unknown to the user because the GUI is changing, so is subject to somewhat less scrutiny, and thus takes 6 minutes as opposed to somewhere over 10 minutes.
If that's true, it is interesting to note that that lesser level of scrutiny is applied even if the Reminders GUI is on a different desktop (as has been the case in my tests*).
UPDATE 3
I tried ad-hoc code-signing this morning with:
codesign --force -v -s - synchRemindersTest.app/Contents/Info.plist synchRemindersTest.app/Contents/PkgInfo synchRemindersTest.app/Contents/Resources/applet.rsrc synchRemindersTest.app/Contents/Resources/Scripts/main.scpt synchRemindersTest.app/Contents/Resources/applet.icns
...and ran the app again, which was a version of TOY 1. Still got the timeout error. I was hoping it would take 40s, like when run from the script editor. When I get time, I'll try it again but manually select a list in the Reminders GUI.
UPDATE 4
I ran the same toy as in update 3 again just now. For the 2 minutes it ran before the timeout, the console was filled with 52,349 lines of mostly this, repeating over and over, which is only the portion of that timespan that matched the search term tccd.
I also have noted that the same unmodified script, run at different times, will succeed in some runs and fail on others. Such as:
TOY 5 (synchRemindersTest5.scpt)
with timeout of 600 seconds
tell application "Reminders"
show list "ToDo Home"
set startt to (get current date)
set allRems to (properties of every reminder whose completed is false)
set endt to (get current date)
set dur to (endt - startt)
set msg to "Got " & (count of allRems) & " reminders in " & dur & " seconds"
tell application "System Events" to display dialog msg giving up after 5
return msg
end tell
end timeout
I ran it repeatedly yesterday, successfully, but I get the timeout when I run it today:
[Jun 08 22:59:51]:~/GoogleDrive/Scripts>osascript synchRemindersTest5.scpt
Got 166 reminders in 287 seconds
[Jun 08 23:06:17]:~/GoogleDrive/Scripts>osascript synchRemindersTest5.scpt
Got 166 reminders in 291 seconds
[Jun 08 23:11:45]:~/GoogleDrive/Scripts>osascript synchRemindersTest5.scpt
Got 166 reminders in 293 seconds
[Jun 08 23:17:46]:~/GoogleDrive/Scripts>osascript synchRemindersTest5.scpt
Got 166 reminders in 300 seconds
[Jun 09 8:23:28]:~/GoogleDrive/Scripts>osascript synchRemindersTest5.scpt
synchRemindersTest5.scpt: execution error: Reminders got an error: AppleEvent timed out. (-1712)
Footnotes
* I've been testing my scripts intentionally with the Reminders app on a different desktop because I have noted in my efforts that GUI scripting is always faster than accessing via the Reminders dictionary. So I wrote 2 methods: GUI and Reminders Dict. The GUI one will run if the open Reminders app is on the desktop (which I keep tucked under the dock). If we're watching Netflix full screen, I have a try/catch to use the slower Reminders Dict access method when the GUI is, as in that case, on a different desktop.
Wrap your script with timeout of 3600 seconds (1 hour). Your script time outs with default time = 2 minutes (120 seconds) per command. So,:
with timeout of 3600 seconds -- or 600 seconds, or as you want
tell application "Reminders"
return (properties of every reminder whose completed is false)
end tell
end timeout
Seing your TOY 4 once again. Man page of osascript says: Any arguments following the script will be passed as a list of strings to
the direct parameter of the ``run'' handler. So, your TOY 4 should be like this:
on run argv -- THIS
with timeout of 3600 seconds
tell application "Reminders"
set allRems to (properties of every reminder whose completed is false)
display dialog "Got " & (count of allRems) & " reminders"
end tell
end timeout
end run -- and THIS
I tried this script in the Terminal, using following command, and it successfully asked for access to Reminders, and worked after access granting. Note quotes as well:
osascript '/Users/123/Desktop/synchRemindersTest.scpt' 'output.json' 'Reminders' 'ToDo'
I did not figure out exactly what was causing the issues, but I repeatedly tried the same exact code with varying results/behaviors, apparently depending on various circumstances. Here are my observations.
Using any of the toy examples, there were 2 running behaviors that seemed to change:
runtime (fastest I could get was near half a minute, but the same code could take over 10 minutes in certain cases - I control-c'd them, so I don't know how long they'd have run)
tccd and other errors in the console (having to do with Apple's mechanism for "transparency, consent, & control" - i.e. the thing that makes those access request popups happen)
I tried running the toy examples above in the following ways:
From Script Editor
Via osascript from the command line
Rewritten as Javascript for Automation (aka "JXA") (from script editor)
As an app, double-clicked
As an app opened from the command line
And I ran those various methods under the following various circumstances (where possible):
Immediately after unlocking the screen
With the Reminders App open on the current desktop
With the Reminders App open off the current desktop
Without manually interacting with the Reminders GUI prior to running
With manually interacting with the Reminders GUI prior to running
With including an applescript instruction to show a list in the Reminders GUI
Without including an applescript instruction to show a list in the Reminders GUI
And there's one other important factor to take into account:
Reminders database size
Apple doesn't actually ever delete anything from the reminders database. I currently have 9,604 completed reminders and 193 incomplete reminders. While exploring this issue, I discovered reminders over a decade old in my reminders database.
I suspect that the issues have more to do with the size of the database than the tccd errors, as I found threads on the Apple Developers forum that describe those errors as mere log noise. I also found posts by developers who note that the ever growing size of the reminders database leads to growing performance issues, and noted that there's no way to really delete entries. Deleted entries are simply labeled as deleted.
I have found that there is no reliable running context that runs both fast and without errors under every circumstance (when you have a large Reminders database). All the ways of executing fail under certain circumstances. Some cases run faster than others, but none ever run in what I would consider a reasonable running time.
I tried code-signing an app version of the toy scripts, explicitly granting entitlements to the reminders data, but according to an app called Taccy, while I could retrieve those entitlements from the files to which they were applied, they did not prevent the tccd errors or make any of the cases run faster. I even tried code-signing a copy of the osascript executable, but apparently it only works for app bundles.
While I could see a difference in runtime under certain circumstances and could avoid tccd errors by doing things in a certain way in some circumstances (all seemingly requiring true manual actions), the runtimes were never significantly improved and errors/failures seemed unavoidable in cases where for example, the screen was locked.
So I concluded that given the size of my reminders database and the fact that I wanted to run this script with the screen locked (e.g. on a cron job), I had to abandon an AppleScript solution. It's impossible to do it predictably & reliably. (I had briefly explored a Siri Automation on an iOS device, but discovered the hoops to jump through to get it to run more than once a day were too annoying.)
So remembering that reminders were(/used to be) stored as ics files in the Library folder. I learned that with the Reminders update in iOS 13 & macOS Catalina, storage of reminders had moved to an sqlite database under ~/Library/Reminders/Container_v1/Stores.
I poked around in the database last night and started to figure things out. I googled some of what I found and found a single google hit to a github repo that had already worked the difficult sqlite stuff out. I ended up with a shell script that reliably retrieves all reminder data (nearly 10k records) in about 1 second!
I have yet to refine it to convert it to JSON and to additionally retrieve anything modified after a certain date, but what I have so far is sufficient to answer this question.
I wrote the shell script in the (unpopular) shell language of tcsh. Feel free to rewrite it in bash or start from the repo I found, which already is in bash (but doesn't retrieve all reminders data):
set REMINDERS_STORES="$HOME/Library/Reminders/Container_v1/Stores";
set SQL_GET_Z_ENT="SELECT Z_ENT FROM Z_PRIMARYKEY WHERE Z_NAME = 'REMCDList'";
foreach DBFILE ( "$REMINDERS_STORES"/Data-*-*.sqlite )
set DB="file:${DBFILE}?mode=ro"
set COUNT=`sqlite3 "$DB" "SELECT COUNT(*) FROM ZREMCDOBJECT WHERE Z_ENT = ($SQL_GET_Z_ENT) AND ZCKIDENTIFIER IS NOT NULL;"`
if ( "$COUNT" > 0 ) then
set REMINDERS_DB="$DB"
endif
end
set Z_ENT_LISTS=`sqlite3 "$REMINDERS_DB" "$SQL_GET_Z_ENT;"`
set YEARZERO=`date -j -f "%Y-%m-%d %H:%M:%S %z" "2001-01-01 0:0:0 +0000" "+%s"`
set NOW=`date "+%s"`
sqlite3 "$REMINDERS_DB" "SELECT strftime('%Y-%m-%dT%H:%M:%S',($YEARZERO + TASK.ZDUEDATE),'unixepoch') as dueDate, TASK.ZPRIORITY AS priority, TASK.ZTITLE1 AS title, LIST.ZNAME1 AS list, TASK.ZNOTES AS notes, TASK.ZCOMPLETED as completed, strftime('%Y-%m-%dT%H:%M:%S',($YEARZERO + TASK.ZCOMPLETIONDATE),'unixepoch') as completionDate, strftime('%Y-%m-%dT%H:%M:%S',($YEARZERO + TASK.ZCREATIONDATE),'unixepoch') as creationDate, TASK.ZDISPLAYDATEISALLDAY as isAllday, strftime('%Y-%m-%dT%H:%M:%S',($YEARZERO + TASK.ZDISPLAYDATEDATE),'unixepoch') as alldayDate, strftime('%Y-%m-%dT%H:%M:%S',($YEARZERO + TASK.ZLASTMODIFIEDDATE),'unixepoch') as modificationDate, TASK.ZFLAGGED as flagged FROM ZREMCDOBJECT TASK LEFT JOIN ZREMCDOBJECT LIST on TASK.ZLIST = LIST.Z_PK WHERE LIST.Z_ENT = $Z_ENT_LISTS AND LIST.ZMARKEDFORDELETION = 0 AND TASK.ZMARKEDFORDELETION = 0 ORDER BY CASE WHEN TASK.ZDUEDATE IS NULL THEN 1 ELSE 0 END, TASK.ZDUEDATE, TASK.ZPRIORITY;"
Here is an example of the output:
2011-11-01T18:30:00|0|Pay the rent|ToDo Home Recurring||1|2011-11-03T13:21:00|2017-09-18T16:59:00|0|2011-11-01T22:30:00|2020-01-04T20:40:00|0
2011-11-05T15:45:00|0|Feed meter|Reminders||1|2011-11-06T15:39:00|2017-09-18T16:59:00|0|2011-11-05T19:45:00|2020-01-04T20:36:00|0
Note, to get the dates in your time zone, as opposed to GMT(/UTC?), append 'localtime', e.g.:
strftime('%Y-%m-%dT%H:%M:%S',($YEARZERO + TASK.ZDUEDATE),'unixepoch', 'localtime')
If I am using the Script Recording and Playback feature on same transaction for instance ME25, multiple times, I am getting cumulative data as part of multiple scripts rather than incremental data.
Explanation :
If I open ME25 details and enter "100-310" as Material and "Ball Bearing" as Short Text and stop the recording, I get the following script, which is expected behavior.
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/ctxtEBAN-MATNR[3,0]").text = "100-310"
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/txtEBAN-TXZ01[4,0]").text = "Ball Bearing"
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/txtEBAN-TXZ01[4,0]").setFocus
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/txtEBAN-TXZ01[4,0]").caretPosition = 12
After this, I restart the recording and type Qty Requested as "100" and delivery date as "21.04.2021" and stop the recording. I get the following script:
session.findById("wnd[0]").maximize
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/ctxtEBAN-MATNR[3,0]").text = "100-310"
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/txtEBAN-TXZ01[4,0]").text = "Ball Bearing"
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/txtEBAN-MENGE[5,0]").text = "100"
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/ctxtRM06B-EEIND[8,0]").text = "21.04.2021"
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/ctxtEBAN-EKGRP[9,0]").setFocus
session.findById("wnd[0]/usr/tblSAPMM06BTC_0106/ctxtEBAN-EKGRP[9,0]").caretPosition = 0
Instead of getting the incremental part that I typed for the second recording instance, I am getting complete script. Is there a way to achieve incremental scripts?
I can reproduce in my SAP GUI 7.60 (whatever screen it is; whatever kind of field it is, I can reproduce even with very simple fields like in a Selection Screen).
It seems that it happens all the time, even if you write your own recorder (a VBS script which mainly uses session.record = True + event handlers). It's due to the fact that SAP GUI sends all the screen events (i.e. the user actions) since the screen was entered, when the user presses a button, a function key, closes a window, or stops the SAP GUI Recorder.
If you write your own recorder, I guess you can save the screen contents when you start the recorder, and when the "change" events occur you may ignore those ones where the new field value has not changed since the recorder started. But that's a lot of work.
I think that it's far more easy to append manually the last lines of the last script to the initial script.
I started using Cypress recently and i noticed that running a test could take 60-80 seconds, but if i go through the same UI "flow" on my browser, it takes 20-30 seconds for me to complete.
Is this normal?
Are there any configurations that are affecting it?
My test is only a few lines long, using only cy.get() and cy.contains().
I have found that one of the causes of slowness of Cypress tests in the GUI is the list of actions unfolding in the Cypress left sidebar.
You will notice that just below the title of each test is the word "⯆ Test" with an arrowhead to its left and below it, all the actions get added as they happen. If you close that list by clicking on the word "Test" the list closes and the tests now run much faster. Now even if you close one, the next test will once more have it open.
To make the default state be closed you need a little hack in the code.
As I have written elsewhere:
In the file Cypress\resources\app\packages\runner\dist\cypress_runner.js look for var Hook = Object in the code. A little below that (line 102918 in version 3.8.3, line 156012 in version 4.5.0) change the isOpen field value from true to false.
Your tests should now run without any slowdown.
Update for versions 6 and up:
In the same file as above, look for the string: this.state === 'failed' and change the line from:
return Boolean(this.state === 'failed' || this.isLongRunning || this.isActive && (this.hasMultipleAttempts || this.isOpenWhenActive) || this.store.hasSingleTest);
to:
return Boolean(this.state === 'failed' /* || this.isLongRunning || this.isActive && (this.hasMultipleAttempts || this.isOpenWhenActive) || this.store.hasSingleTest */);
Cypress test are much slower than unit test and that's normal. The difference between UI and headless run may caused by cypress initialising between command and test.
To reduce time needed to pass test avoid cy.wait(, instead use e.g. cy.get( instead.
Also you can try how long takes to run with --headless --browser chrome flags.
Cypress has to run a proxy on 3rd party browsers so that they can record the requests being sent and received. This can slow down load times considerably.
The solution I used was to run my tests in their provided electron browser. e.g.
cypress run -s [your spec file] --headed -b electron
--headed - Shows the electron window so you can know what's happening
-b electron - Use the electron browser
I have a question about Xamarin Test Cloud, hope someone can point me in the right direction.
When a user taps a button in my app, a process runs for around 30 minutes. I added a Unit Test project and it runs perfectly in the emulator.
However, I need to test it in real devices, so I decided to use Xamarin Test Cloud. When I run the test there, it doesn't complete it. As I said, it should take 30 minutes but the test finishes almost immediately.
Here is the code of my Test:
[Test]
[Timeout(Int32.MaxValue)]
public async void Optimize()
{
await Task.Run(async() =>
{
app.Screenshot("Start " + DateTime.Now);
app.Tap(x => x.Marked("btnOptimize"));
await Task.Delay(120000);
app.Screenshot("End " + DateTime.Now);
}
}
If I run the test in the emulator, the screenshot names are (for instance) "Start 12:00:00" and "End 12:30:00" respectively (so it means that it runs for 30 minutes, as expected). However, in Test Cloud I get (for instance) "Start 12:00:00" and "End 12:02:00", which means that the test runs for only 2 minutes. But that's because I added the delay. Without the delay, it will run for only 5 seconds.
Is that what I need? I can add 1800000 so the test can be completed in 30 minutes, but what if I don't know the time?
Thank you and sorry if it's a basic question
Something like this should do the job:
[Test]
public async void Optimize()
{
app.Screenshot("Start");
app.Tap("btnOptimize");
app.WaitForElement ("otherButton", "Timed out waiting for Button", new TimeSpan(0,30,0));
app.Screenshot("End");
}
Where "otherButton" becomes visible when the task is done. There are other Wait methods available in the API.
But, note that the Xamarin Test Cloud has a thirty minute maximum per test by default. That default can be modified by contacting Xamarin Test Cloud support.
Also, it is not a good practice to include non-deterministic information or any information that may vary per device or run in your screen shot titles. When you run on more than one device the test report steps and screenshots are collated partially by the screen shot titles so they should match across devices for best results.
While I have never tried a timeout length of 30 minutes, the Calabash allows you to wait for a condition using wait_for):
The following snippet is an example of how to use wait_for to detect the presence of a button on the screen:
wait_for(timeout: 60, timeout_message: "Could not find 'Sign in' button") do
element_exists("button marked:'Sign in'")
end
Ref: https://docs.xamarin.com/guides/testcloud/calabash/working-with/timeouts/
Just an FYI: 30 minutes is a really long time for a mobile device to be "background processing" without user interface interaction, if you are targeting iOS/Apple Store, this death sentence in getting Apple submission approval as they will never wait that long for an app to process something....
You need to identify tap by Id and add WaitForElement(first argument is query you want to wait on) in correct syntax like given below. It should work for you.
app.Screenshot("Start " + DateTime.Now);
app.WaitForElement(x => x.Id("btnOptimize"),"Optimization timed out",new System.TimeSpan(0,30,0),null,null);
app.Tap(x => x.Id("btnOptimize"));
app.Screenshot("End " + DateTime.Now);
I'm using the Selenium WebDriver and Ruby to perform some automation and I ran into a problem of a captcha around step 3 of a 5 step process.
I'm throwing all the automation in a rake script so I'm wondering is there a command to pause or break the script running temporarily until I enter data into the captcha and then continue running on the next page.
To build on seleniumnewbie's answer and assuming that you have access to the console the script is running on:
print "Enter Captcha"
captchaTxt = gets.chomp
yourCaptchaInputWebdriverElement.send_keys captchaTxt
If you just want to pause and enter the captcha in your browser, you can just have it prompt at the console to do that very thing and it'll just sit there.
print "Enter the captcha in your browser"
gets
You could also set the implicit wait to decently long period of time so that Selenium would automatically see the next page and move out. However, this would leave the important Captcha step (for documenting / processes sake) out of your test unless you're pretty anal with your commenting.
Since this is an actively participating test requiring user input I would say that making the tester press "enter" on the console is the way to go.
Since you are writing the test in a script, all you need to do is add a sleep in your test i.e. 'sleep 100' for example.
However, it is bad to add arbitrary sleeps in tests. You can also do something like "Wait for title 'foo'" where 'foo' is the title of the page in Step 4. It need not be title, it can be anything, but you get the idea. Wait for something semantic which indicates that step 3 is done and step 4 is ready to start.
This way, its more targeted wait.
This has been implemented in JAVA, but similar technique.Your solution could be found here