Run remote backup on regulary base - macos

I wrote a shell script which checks if a remote server is reachable.
If it is reachable, it runs via ssh a command at a different server.
After that, it does some rsync tasks.
Everything works well, so I wrote a plist-files to run the whole staff every 4 hours.
What's work:
The script is triggered as expected (I saw it at stdout-file).
launchctl start myjob works fine too (I saw it at stdout-file).
My problem:
If the job starts during the macbook sleeps, the ping works well, but the ssh command doesn't come back. The command is executed at the server, but the script stops there.
If I kill the process with the ssh-command the script finish well.
What can I do to get more information. What could be the error? What could be another solution (Timemachine is not an option, because I want a general readable backup)?
Mac OS BigSur 11.4 M1-Chip
Here my plist-files
?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>de.stanetz.backup</string>
<key>Program</key>
<string>/usr/local/sbin/remoteBackup.nsh</string>
<key>RunAtLoad</key>
<false/>
<key>StandardOutPath</key>
<string>/var/log/launch-stdout.log</string>
<key>StandardErrorPath</key>
<string>/var/log/launch-stderr.log</string>
<key>Debug</key>
<true/>
<key>StartCalendarInterval</key>
<array>
<dict>
<key>Hour</key>
<integer>9</integer>
<key>Minute</key>
<integer>15</integer>
</dict>
</array>
<key>ExitTimeOut</key>
<integer>360</integer>
<key>TimeOut</key>
<integer>360</integer>
</dict>
</plist>
Update: I changed the ssh command to ssh -vv pooh 'echo "this runs remotely"' >/tmp/so-debug.log 2>&1. Which works fine.
If I call debug1: Sending command: /usr/local/sbin/handleBackupPlatte.nsh open via ssh (which runs 10 seconds), the complete shell-script finished, but I found in the log:
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Killed by signal 15.
The duration of time till the remote-script finished seems to be the problem.
#!/bin/bash
echo "this runs remotely"
sleep 10
echo "for 10 seconds"
shows the effect. But I can't make the script faster :-/ Any ideas, except to send the ssh command to background?
Update 2: It seems that every network call which needs 10 seconds will hang and never finished if the mac book is in sleep-state.

I switch to a different approach. I install sleepwatcher which notify if a macbook is wake up. Furthermore, I combined this with a script which ensures that not on every wake up the backup is running.
Here the script
#!/bin/bash
#set -x
set -eu
## RETURN_CODES
# 11 -> BACKUP_ALREADY_RUNNING
# 12 -> NIGHT -> No Backup
# 13 -> Last backup is too young.
#VARIABLES
LOCK_FILE="/tmp/.backup.lock"
LAST_SUCCESS_FILE="/var/log/.backup.last_success"
RUN_CONTROL="/var/log/throttle.log"
MIN_BACKUP_AGE=240
cleanup() {
exitCode=$?
date +"Exit Throttle at %d.%m.%Y %H:%M:%S with $exitCode" >> ${RUN_CONTROL}
if [[ "$exitCode" -ne 11 ]]; then
rm ${LOCK_FILE}
fi
echo "$(tail -n 100 ${RUN_CONTROL})" > ${RUN_CONTROL}
exit $exitCode
}
trap cleanup EXIT
test -e ${LOCK_FILE} && echo "${LOCK_FILE} exists" && exit 11
touch ${LOCK_FILE}
hour=$(date +%H)
if [ "$hour" -gt 22 ] || [ "$hour" -lt 7 ]; then
exit 12
fi
test -e "$(find ${LAST_SUCCESS_FILE} -mmin -${MIN_BACKUP_AGE})" && exit 13 || true
/usr/local/sbin/remoteBackup.nsh
echo "$(tail ${LAST_SUCCESS_FILE})" > ${LAST_SUCCESS_FILE}
date >> ${LAST_SUCCESS_FILE}
and the /Library/LaunchDaemons/de.bernhard-baehr.sleepwatcher.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>de.bernhard-baehr.sleepwatcher</string>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/sbin/sleepwatcher</string>
<string>-V</string>
<string>-w /usr/local/sbin/throttledBackup.sh</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<key>StandardOutPath</key>
<string>/var/log/sleepwatcher-stdout.log</string>
<key>StandardErrorPath</key>
<string>/var/log/sleepwatcher-stderr.log</string>
</dict>
</plist>

Related

launchctl status returns 2

I'm trying to write a launchd script that will run a Python script every hour.
I created the following file in ~/Library/LaunchAgents/
com.b00tahead.engage-check.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.b00tahead.engage-check</string>
<key>ProgramArguments</key>
<array>
<string>/Library/Frameworks/Python.framework/Versions/3.5/bin/python3</string>
<string>Projects/engage_scripts/engage_check.py</string>
</array>
<key>StartInterval</key>
<integer>3600</integer>
</dict>
</plist>
I then run:
launchctl load ~/Library/LaunchAgents/com.b00tahead.engage-check.plist
launchctl start com.b00tahead.engage-check
When I run launchctl list it shows the process as:
PID Status Label
- 2 com.b00tahead.engage-check
I'm not exactly sure what that return code indicates. I can run the script just fine through the terminal as python3 Projects/engage_scripts/engage_check.py
The Python script writes some data to a .txt file on my desktop.
This is what launchctl list com.b00tahead.engage-check returns:
{
"LimitLoadToSessionType" = "Aqua";
"Label" = "com.b00tahead.engage-check";
"TimeOut" = 30;
"OnDemand" = true;
"LastExitStatus" = 512;
"Program" = "/Library/Frameworks/Python.framework/Versions/3.5/bin/python3";
"ProgramArguments" = (
"/Library/Frameworks/Python.framework/Versions/3.5/bin/python3";
"Projects/engage_scripts/engage_check.py";
);
};
My guess is that the argument in ProgramArguments should have an absolute path (not relative such as Projects/...).
In case of error, system log should be checked, e.g.
tail -f /var/log/system.log
or custom log can be used by specifying these lines:
<key>StandardOutPath</key>
<string>/var/log/myjob.log</string>
<key>StandardErrorPath</key>
<string>/var/log/myjob.log</string>
See: Debugging launchd Jobs section of Creating Launch Daemons and Agents page.

At my end with launchd

Hi all and thanks for reading this. I have the below launchd plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.me.sendip</string>
<key>Program</key>
<string>/Users/me/Documents/sendip.sh</string>
<key>StartInterval</key>
<integer>60</integer>
<key>StandardErrorPath</key>
<string>/Users/me/Desktop/sendiperr.log</string>
<key>StandardOutPath</key>
<string>/Users/me/Desktop/sendip.log</string>
</dict>
I also have the below script
/Users/me/Documents/Scripts/ip.sh | mail -s "Laptop ip" me#gmail.com
I have tried troubleshooting with all the tools here. The script fires with an empty log file on boot but after that nothing.
Someone please help. I am totally stumped and I am sure its something silly. Much appreciated.
It turns out launchd abandons the sub processes of the script I was trying to run. this is a common problem as it turns out. Found the answer here.
Hmmm. I run a script at hourly intervals using the following plist in 10.9.5.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>Label</key>
<string>org.myorg.mytask</string>
<key>ProgramArguments</key>
<array>
<string>/usr/bin/php</string>
<string>/path/to/myscript.php</string>
</array>
<key>QueueDirectories</key>
<array/>
<key>StartCalendarInterval</key>
<dict>
<key>Minute</key>
<integer>5</integer>
</dict>
<key>WatchPaths</key>
<array/>
</dict>
</plist>
That runs every hour at 5 minutes past the hour.
I think for you're looking to run once a minute, right? So your version would be:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>Label</key>
<string>com.me.sendip</string>
<key>ProgramArguments</key>
<array>
<string>/Users/me/Documents/sendip.sh</string>
</array>
<key>QueueDirectories</key>
<array/>
<key>StartInterval</key>
<integer>60</integer>
<key>WatchPaths</key>
<array/>
</dict>
</plist>
You may need to start the service by running this command in Terminal: (it will require your password)
sudo launchctl load com.me.sendip
Edit: Here is a script with more explicit logging:
#!/bin/bash
OUTFILE='/Library/Logs/com.whatever.sendip.log'
ERRFILE='/Library/Logs/com.whatever.sendip.err.log'
SCRIPT='/Users/me/Documents/Scripts/ip.sh'
TARGET_EMAIL='me#gmail.com'
TARGET_SUBJECT='Laptop ip'
VERBOSE=1
if [ "$VERBOSE" -ne 0 ]; then
date >>"$OUTFILE"
date >>"$ERRFILE"
echo -n 'Current user: ' >>"$OUTFILE"
whoami >>"$OUTFILE"
fi
INFO=$("$SCRIPT" 2>>"$ERRFILE")
echo "$INFO" | mail -s "$TARGET_SUBJECT" "$TARGET_EMAIL" >>"$OUTFILE" 2>>"$ERRFILE"
ERROR_CODE="$?"
if [ "$ERROR_CODE" -ne 0 ]; then
echo "Result: Error Code $ERROR_CODE" >>"$OUTFILE"
echo "Info: $INFO" >>"$OUTFILE"
elif [ "$VERBOSE" -ne 0 ]; then
echo "Result: Success" >>"$OUTFILE"
echo "Info: $INFO" >>"$OUTFILE"
fi

Trouble getting a launchctl login script to run on Mac OS X

I've been reading this topic: Running script upon login mac and trying to follow its guidelines for using launchctl to create a bash shell script that runs automatically at login. I just CAN'T GET IT TO WORK! The shell script I want to launch is as follows:
#!/bin/bash
#
# Shell script to start the RobotFramework Auto-Started Servers.
#
# Copyright (c) 2014 Texas Instruments, Inc.
#
#-------------------------------------------------------------------------------
#
#
# Start the dispatcher as an independent background task with its own virtual screen
#
cd /Users/epsqainfprod/RobotFramework/Extensions/Dispatcher
screen -S Dispatcher -d -m ./StartDispatcher.sh
#
# Start the RunProcServer
#
cd /Users/epsqainfprod/RobotFramework/Extensions/RunProcess
screen -S RunProcServer -d -m ./RunProcServer.sh
#
# -------------------------------- End of file --------------------------------
If I invoke this from the command line it works fine, creating two independently running, screen processes. I created a .plist file for this, following the instructions from the referenced post:
cat ~/Library/LaunchAgents/com.user.loginscript.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.user.loginscript</string>
<key>Program</key>
<string>/Users/epsqainfprod/TEMP/AutoStartScripts.sh</string>
<key>RunAtLoad</key>
<true/>
</dict>
where the above shell script is in the file /Users/epsqainfprod/TEMP/AutoStartScripts.sh. I've checked system.log and all it shows is:
Dec 2 08:05:49 epsqainf5lab15.itg.ti.com login[2109]: USER_PROCESS: 2109 ttys000
Dec 2 08:05:49 epsqainf5lab15.itg.ti.com login[2109]: DEAD_PROCESS: 2109 ttys000
Dec 2 08:07:33 epsqainf5lab15.itg.ti.com login[2117]: USER_PROCESS: 2117 ttys000
Dec 2 08:07:33 epsqainf5lab15.itg.ti.com login[2117]: DEAD_PROCESS: 2117 ttys000
So I'm completely stumped on this. Does anyone have any suggestions how to get this working?
Seems there were two problems with my approach:
The "screen" command only seems to work when you're running it from a command with ancestry to a terminal screen.
If "screen" worked, then it would return. If both "screen" steps worked then the AutoStartScripts.sh would terminate.
My solution, now working fine, was to create two shell scripts:
AutoStartDispatcher.sh
AutoStartRunProcServer.sh
and then create two .plist files, one pointing to each of the two .sh files:
RF.AutoLaunch.Dispatcher.plist
RF.AutoLaunch.RunProcServer.plist
Then I went through the "launchctl load <.plist file path>" step for each of the .plist files.

Inside shell script: write Python output to file works from terminal, "Input stream is empty" when running from launchd

I wrote a short shell script to write the contents of an Evernote note to a file with Geeknote, which runs under python:
#!/bin/bash
/usr/bin/python /Users/me/Git/geeknote/geeknote.py find --search "Hobbies To Do" > /dev/null
/usr/bin/python -u /Users/me/Git/geeknote/geeknote.py show "Hobbies To Do" | tee /Users/me/Programming/output.txt
The script works fine when I run it from terminal (it writes the output of the second python command to output.txt), although ideally I'd like the script to run periodically on its own. So I tried creating a .plist/LaunchAgent to run the script, but when I do that the only thing that gets written to output.txt is Input stream is empty...can't figure out why. Right now my .plist looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.me.geektoolcron</string>
<key>ProgramArguments</key>
<array>
<string>/Users/me/Programming/script.sh</string>
</array>
<key>StartInterval</key>
<integer>10</integer>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
You could try replacing the second python command with something like echo a or replacing tee with a redirection. A script like this worked for me with the same plist:
#!/bin/bash
date | tee /tmp/f
launchd also supports directing stdout to a file:
<key>StandardOutPath</key>
<string>/Users/me/Programming/output.txt</string>

pandoc via launchd -- getting error: "pdflatex not found. pdflatex is needed for pdf output

I am trying to generate a pdf document from a markdown document every 90 seconds. I have set up a task in launchd to handle this for me. Unfortunately, I am getting an error announcing that pdflatex can't be found even though I can invoke it from the command line. Here is the entire contents of ddd_publisher.sh, which is the file that launchd runs every 90 seconds:
/usr/local/bin/pandoc -o /Users/Jon/dev/intercontinental/ddd.pdf /Users/Jon/dev/intercontinental/ddd.ddd
The program runs every 90 seconds, but it writes the following line to stderr:
pandoc: pdflatex not found. pdflatex is needed for pdf output.
If I type pdflatex -v at the command line it tells me that it is installed:
pdfTeX 3.1415926-2.4-1.40.13 (TeX Live 2012)
Here is my launchd plist file:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>ddd.intercontinental.publisher</string>
<key>ProgramArguments</key>
<array>
<string>/Users/Jon/dev/intercontinental/ddd_publisher.sh</string>
</array>
<key>StandardErrorPath</key>
<string>/Users/Jon/dev/intercontinental/ddd.stderr</string>
<key>StandardOutPath</key>
<string>/Users/Jon/dev/intercontinental/ddd.stdout</string>
<key>StartInterval</key>
<integer>90</integer>
The solution was to add the path for pdflatex to the bash script, as follows:
#!/bin/bash
PATH=$PATH:/usr/texbin
/usr/local/bin/pandoc -o /Users/Jon/dev/intercontinental/ddd.pdf /Users/Jon/dev/intercontinental/ddd.ddd

Resources