Ive got a script that checks if a process is running or not, using ps -ef and some grep. If the process is running, it does nothing. If the process isnt running, it restarts the process, and then sends an email to me stating that the process died.
This script currently runs ever 5 minutes from a admin account's crontab.
Peusdocode + problem code:
#!/bin/sh
# declare a ton of environment variables.
ATONOFVARS=/lots/of/qualified/paths
# declare relevant logging functions
loggingFunction()
# declare failure message to pass into mail (real code)
PAGEMESSAGE="MyServer: process not found in ps -ef. Attempting to Restart... Please check log file in /tmp"
PAGESUBJECT="ProcessHealthCheck.sh - MyServer: process not found in ps -ef"
EMAILLIST="my.email#mycorp.com"
# do the actual check (back to peusdo code)
if [ $(ps -ef | grep process | grep adminAccount | grep -v grep | wc -1) ]
restartProcess()
FAILURE=1
fi
# log the failure and send an email to me (real code)
if [ $FAILURE -eq 1 ]
then
logMsg ""
logMsg "The process is not detected in ps -ef!"
logMsg "Restarting..."
logMsg ""
#send emails out
echo $PAGEMESSAGE | mail -s $PAGESUBJECT $EMAILLIST
fi
To test this, I deliberately killed the PID of my process I want monitored, so this cronjob would have to restart it, I also wanted to test the email capability. The script restarts my process exactly like I want it, and I get the email.
However the email says in the message body: "MyServer: process not found in
the entire listing of ps -ef. Attempting to Restart... Please check log file in /tmp". Also, the addresses that the email was sent to, are lines from ps -ef as well. As in my email had 2000 recipients. Sample addresses: grep#myServer.mydomain.com, process#myServer.mydomain.com
Does anyone know whats going on? Why is mail executing a command that is in a string? Since finding this out, I have changed the string to remove any possibility of a unix command.
Related
I am trying to run a command in shell script and would like to exit it if the processing logs (not sure what you call the logs that are outputted on terminal while the task is running) contains the string "INFO | Next session will start at"
I tried using grep but because the string "INFO | Next session will start at" is not in a stdout it does not detect while the command is running.
The specific command I'm running is below
pipenv run python3 run.py --config accounts/user/config.yml
By 'processing logs' I mean the log output before the stdout is displayed in the terminal.
...
[D 211127 10:07:12 init:400] atx-agent version 0.10.0
[D 211127 10:07:12 init:403] device wlan ip: route ip+net: no such network interface
[11/27 10:07:12] INFO | Time delta has set to 00:11:51.
[11/27 10:07:13] INFO | Kill atx agent.
[11/27 09:59:32] INFO | Next session will start at: 10:28:30 (2021/11/27).
[11/27 09:59:32] INFO | Time left: 00:28:57.
I am trying to do this because the yml file I'm trying to run has a limit on what time you can execute it, and I would like to exit the task if the time is not met.
I tried to give as much context but if there's something missing please let me know.
This may work:
pipenv run python3 run.py --config accounts/user/config.yml |
sed "/INFO | Next session will start at/q"
sed prints the piped input, until it matches the expression and quits (q). The program will receive SIGPIPE (broken pipe) when it tries to continue writing, and (likely) exit. It's the same as what happens when you do something like find | head.
You could also use kill in a shell wrapper:
sh -c 'pipenv run python3 run.py --config accounts/user/config.yml |
{ sed "/INFO | Next session will start at/q"; kill -- -$$; }'
Notes:
The program may print a different log if stdout is not a terminal.
If you want to match a literal string, you could use grep -Fm 1 PATTERN, but other log output will be hidden. grep fails if no match, which can be useful.
This will work any shell, including zsh. zsh or bash can also be used for the kill wrapper.
There are other approaches. This thread focuses on tail, but is a useful reference: https://superuser.com/questions/270529/monitoring-a-file-until-a-string-is-found
Is it possible to listen for a specific notification on Mac and act upon it using Automator?
I regularly use an app that runs a background job then sends a notification when it's finished. The app stays open after the job is finished so I'd like to use Automator to quit the app when the notification is received.
If it's not possible in Automator is there another way I could do this?
More context: the app is actually launched by a Folder Action created using Automator. It detects when a specific SD card is inserted and runs a backup app on that SD card. So maybe there's something I can add to that Folder Action workflow that can detect the notification?
While it's certainly possible to query the sqlite database containing notifications in macOS, it seems to me like an unnecessarily complicated route, and I would first try the following...
In your workflow, add a 'Run Shell Script' action at the end, containing something like this:
while [ $(ps -e | grep "[N]ameOfBackgroundProcess" | wc -l) -gt 0 ]; do
sleep 3
done
killall "NameOfAppToQuit" 1&>2 /dev/null
The while loop checks whether the background job is still running.
ps -e lists all running processes.
grep "[N]ameOf..." gets all lines containing the name. Brackets around the first letter excludes the grep process itself from the output.
wc -l counts the lines.
-gt 0 checks if the number is greater than zero.
When the loop is done, that means the process has exited so we quit the app with killall.
As for the notification route...
I haven't figured everything out, but this might give you a head start:
#!/usr/bin/env bash
# Get the directory of the Notification Center database (works for me in Big Sur):
db_dir=$(lsof -p $(ps aux | grep -m1 usernoted | awk '{ print $2 }')| awk '{ print $NF }' | grep 'db2/db$' | xargs dirname)
# Get the app_id:
app_id=$(sqlite3 "$db_dir"/db 'SELECT app_id FROM app WHERE identifier="com.example.identifier";')
# Get relevant records:
sqlite3 "$db_dir"/db "SELECT * FROM record WHERE app_id='$app_id';"
# And this is where I leave you.
To explore the database in a GUI, try https://sqlitebrowser.org/
I have a shell script (script.sh) in which I run a simple python web scraper
python3 script.py
As I run this shell script in a cron job, I want to be notified by email if something goes wrong. I found this code on stackoverflow, which I put at the bottom of my .sh script:
./script.sh 2>&1 | tee output.txt | mail -s "Script log" email#address.com
The problem is that the script now seems to be looping; where it should only take a few seconds, it now takes a few minutes to run and I receive 10-20 emails in my mailbox. The content of these emails differs, most of the times the emails are empty, but sometimes they contain messages such as:
./script.sh: 4: ./script.sh: Cannot fork
or:
mail: Null message body; hope that's ok
I'm not sure what goes wrong here. What can I do to fix this?
The script runs, runs the python, then calls itself again, until it runs out of resources and fails to fork.
The .sh should contain:
python3 script.py 2>&1 | tee output.txt | mail -s "Script log" email#address.com
Assuming you actually want to create output.txt in whatever folder pwd is... otherwise you could leave that out entirely.
Alternatively you could just configure your MAILTO in your crontab (see https://www.cyberciti.biz/faq/linux-unix-crontab-change-mailto-settings/)
I have a java program that stops often due to errors which is logged in a .log file. What can be a simple shell script to detect a particular text in the last/latest line say
[INFO] Stream closed
and then run the following command
java -jar xyz.jar
This should keep on happening forever(possibly after every two minutes or so) because xyz.jar writes the log file.
The text stream closed can arrive a lot of times in the log file. I just want it to take an action when it comes in the last line.
How about
while [[ true ]];
do
sleep 120
tail -1 logfile | grep -q "[INFO] Stream Closed"
if [[ $? -eq 1 ]]
then
java -jar xyz.jar &
fi
done
There may be condition where the tailed last log "Stream Closed" is not the real last log and the process is still logging the messages. We can avoid this condition by checking if the process is alive or not. If the process exited and the last log is "Stream Closed" then we need to restart the application.
#!/bin/bash
java -jar xyz.jar &
PID=$1
while [ true ]
do
tail -1 logfile | grep -q "Stream Closed" && kill -0 $PID && sleep 20 && continue
java -jar xyz.jar &
PID=$1
done
I would prefer checking whether the corresponding process is still running and restart the program on that event. There might be other errors that cause the process to stop. You can use a cronjob to periodically (like every minute) perform such a check.
Also, you might want to improve your java code so that it does not crash that often (if you have access to the code).
i solved this using a watchdog script that checks directly (grep) if program(s) is(are) running. by calling watchdog every minute (from cron under ubuntu), i basically guarantee (programs and environment are VERY stable) that no program will stay offline for more than 59 seconds.
this script will check a list of programs using the name in an array and see if each one is running, and, in case not, start it.
#!/bin/bash
#
# watchdog
#
# Run as a cron job to keep an eye on what_to_monitor which should always
# be running. Restart what_to_monitor and send notification as needed.
#
# This needs to be run as root or a user that can start system services.
#
# Revisions: 0.1 (20100506), 0.2 (20100507)
# first prog to check
NAME[0]=soc_gt2
# 2nd
NAME[1]=soc_gt0
# 3rd, etc etc
NAME[2]=soc_gp00
# START=/usr/sbin/$NAME
NOTIFY=you#gmail.com
NOTIFYCC=you2#mail.com
GREP=/bin/grep
PS=/bin/ps
NOP=/bin/true
DATE=/bin/date
MAIL=/bin/mail
RM=/bin/rm
for nameTemp in "${NAME[#]}"; do
$PS -ef|$GREP -v grep|$GREP $nameTemp >/dev/null 2>&1
case "$?" in
0)
# It is running in this case so we do nothing.
echo "$nameTemp is RUNNING OK. Relax."
$NOP
;;
1)
echo "$nameTemp is NOT RUNNING. Starting $nameTemp and sending notices."
START=/usr/sbin/$nameTemp
$START 2>&1 >/dev/null &
NOTICE=/tmp/watchdog.txt
echo "$NAME was not running and was started on `$DATE`" > $NOTICE
# $MAIL -n -s "watchdog notice" -c $NOTIFYCC $NOTIFY < $NOTICE
$RM -f $NOTICE
;;
esac
done
exit
i do not use the log verification, though you could easily incorporate that into your own version (just change grep for log check, for example).
if you run it from command line (or putty, if you are remotely connected), you will see what was working and what wasnt. have been using it for months now without a hiccup. just call it whenever you want to see what's working (regardless of it running under cron).
you could also place all your critical programs in one folder, do a directory list and check if every file in that folder has a program running under the same name. or read a txt file line by line, with every line correspoding to a program that is supposed to be running. etcetcetc
A good way is to use the awk command:
tail -f somelog.log | awk '/.*[INFO] Stream Closed.*/ { system("java -jar xyz.jar") }'
This continually monitors the log stream and when the regular expression matches its fires off whatever system command you have set, which is anything you would type into a shell.
If you really wanna be good you can put that line into a .sh file and run that .sh file from a process monitoring daemon like upstart to ensure that it never dies.
Nice and clean =D
I use Ubuntu and am trying to write a script that makes the following:
-test if an audio stream works
-if not, send an email.
I have tried the following code (running as a cron job every 10 minutes), which 'works' if I supply the wrong pw e.g.(it sends an email then), but does nothing if the actual server is down (tested by killing the server). any ideas on how to fix the script?
Thanks in advance!
#!/bin/bash
#servertest.sh
username=user1
password=xyz
url="http://wwww.streamingaudioserver.com -passwd $password -user $username"
mplayer $url &
sleep 5
test=$(pgrep -c mplayer)
if [ $test = 0 ]; then
#server is down!
mailfile="downmail.txt"
/usr/sbin/ssmtp test#maildomain.com < "/home/test/$mailfile"
fi
killall mplayer
sleep 5
exit
Your problem is in this line:
$mailfile="downmail.txt"
remove the dollar sign and that should do it.
You should be getting error messages in your cron log or emails to the crontab owner complaining about a command not found or no such file.
Edit:
Does your script work if run from the command line (with the stream down) rather than cron?
Try using set -x (or #!/bin/bash -x) in the script to turn on tracing or use echo "PID: $$, value of \$test: $test" > /tmp/script.out after the assignment to see if you're getting the zero you're expecting.
Also, try an ssmtp command outside the if to make sure it's working (but I think you already said it is under some circumstances).
Try your script without ever starting mplayer.