I am trying to run a command in shell script and would like to exit it if the processing logs (not sure what you call the logs that are outputted on terminal while the task is running) contains the string "INFO | Next session will start at"
I tried using grep but because the string "INFO | Next session will start at" is not in a stdout it does not detect while the command is running.
The specific command I'm running is below
pipenv run python3 run.py --config accounts/user/config.yml
By 'processing logs' I mean the log output before the stdout is displayed in the terminal.
...
[D 211127 10:07:12 init:400] atx-agent version 0.10.0
[D 211127 10:07:12 init:403] device wlan ip: route ip+net: no such network interface
[11/27 10:07:12] INFO | Time delta has set to 00:11:51.
[11/27 10:07:13] INFO | Kill atx agent.
[11/27 09:59:32] INFO | Next session will start at: 10:28:30 (2021/11/27).
[11/27 09:59:32] INFO | Time left: 00:28:57.
I am trying to do this because the yml file I'm trying to run has a limit on what time you can execute it, and I would like to exit the task if the time is not met.
I tried to give as much context but if there's something missing please let me know.
This may work:
pipenv run python3 run.py --config accounts/user/config.yml |
sed "/INFO | Next session will start at/q"
sed prints the piped input, until it matches the expression and quits (q). The program will receive SIGPIPE (broken pipe) when it tries to continue writing, and (likely) exit. It's the same as what happens when you do something like find | head.
You could also use kill in a shell wrapper:
sh -c 'pipenv run python3 run.py --config accounts/user/config.yml |
{ sed "/INFO | Next session will start at/q"; kill -- -$$; }'
Notes:
The program may print a different log if stdout is not a terminal.
If you want to match a literal string, you could use grep -Fm 1 PATTERN, but other log output will be hidden. grep fails if no match, which can be useful.
This will work any shell, including zsh. zsh or bash can also be used for the kill wrapper.
There are other approaches. This thread focuses on tail, but is a useful reference: https://superuser.com/questions/270529/monitoring-a-file-until-a-string-is-found
Related
I have this need to collect\log all the command lines that were used to start a process on my machine during the execution of a Perl script which happens to be a test automation script. This Perl script starts the executable in question (MySQL) multiple times with various command lines and I would like to inspect all of the command lines of those invocations. What would be the right way to do this? One possibility i see is run something like "ps -aux | grep mysqld | grep -v grep" in a loop in a shell script and capture the results in a file but then I would have to do some post processing on this and remove duplicates etc and I could possibly miss some process command lines because of timing issues. Is there a better way to achieve this.
Processing the ps output can always miss some processes. It will only capture the ones currently existing. The best way would be to modify the Perl script to log each command before or after it executes it.
If that's not an option, you can get the child pids of the perl script by running:
pgrep -P $pid -a
-a gives the full process command. $pid is the pid of the perl script. Then process just those.
You could use strace to log calls to execve.
$ strace -f -o strace.out -e execve perl -e 'system("echo hello")'
hello
$ egrep ' = 0$' strace.out
11232 execve("/usr/bin/perl", ["perl", "-e", "system(\"echo hello\")"], 0x7ffc6d8e3478 /* 55 vars */) = 0
11233 execve("/bin/echo", ["echo", "hello"], 0x55f388200cf0 /* 55 vars */) = 0
Note that strace.out will also show the failed execs (where execve returned -1), hence the egrep command to find the successful ones. A successful execve call does not return, but strace records it as if it returned 0.
Be aware that this is a relatively expensive solution because it is necessary to include the -f option (follow forks), as perl will be doing the exec call from forked subprocesses. This is applied recursively, so it means that your MySQL executable will itself run through strace. But for a one-off diagnostic test it might be acceptable.
Because of the need to use recursion, any exec calls done from your MySQL executable will also appear in the strace.out, and you will have to filter those out. But the PID is shown for all calls, and if you were to log also any fork or clone calls (i.e. strace -e execve,fork,clone), you would see both the parent and child PIDs, in the form <parent_pid> clone(......) = <child_pid> so then you should hopefully then have enough information to reconstruct the process tree and decide which processes you are interested in.
I am using BSD grep! (Different from UNIX grep)
grep -e '22T1[2-4]' nagoya_all.csv -> nagoya12_14.csv
When I execute this command, nagoya12_14.csv is successfully created. However, on terminal is doesn't prompt me for a new command.
Why does this happen?
How can i check if this command is still running?
Take out the - in front the redirect sign > and you should be good. When you have - at the end of a grep command, it will try to read from stdin, which waits for user input.
I'm executing
eb deploy my_env,
After which quite a bit of text is shown in command line, and I would like to grep for "update completed successfully" Upon Completion of the above command, and if not found return an exit status of 1.
I was wondering how this is possible.
(If you're interested in why, I've found that eb deployments despite a deploy failure return an exit status of 0)
This is a textbook case of what pipes are used for:
eb deploy my_env | grep "update completed successfully"
If you want to suppress the output of grep, use the -q flag. The result will act just as if eb deploy had an error code that actually matched your needs:
eb deploy my_env | grep -q "update completed successfully"
As you mention in your comments, eb deploy my_env can actually be any other pipeline of commands that contains "update completed successfully" somewhere in the final output. The return value for the entire pipeline will be the return value of grep, being the last command.
This type of piping is actually a fundamental principle of UNIX design. UNIX commands are ideally small blocks that perform one single function and do it well. The idea is that it is much easier to pipe together some number of robust modules than to write a single very complex program that does everything.
Interesting Update
You can actually see the raw output from eb deploy and still use grep to determine the return code. The simplest way (which better illustrates chained pipes) is to use process substitution, which is available in bash, ksh and zsh:
eb deploy my_env | tee >(cat) | grep -q "update completed successfully"
tee will duplicate the output to a file and stdout. stdout will be piped to grep as before. Instead of specifying a file, we use a sub-process >(cat), which will just print the output to the command line.
For other, more portable methods, refer to this post on Unix/Linux stack exchange for more info.
Grep returns an exit code. To use only the exit code without outputting the match, use the following:
if <command> | grep -q "<text>"; then ...
-q is for quiet mode.
Pipe the output to a file. Then grep that output.
eb deploy my_env > deploy_results.txt
grep 'update completed successfully' deploy_results.txt
I have very long running shell script ( runs more than 24 hours) .
It is very simple script. It just reads xml file from a dir and perform sed operation of the content of the file. There are 1 millions xml files in the dir.
My script name is like runDataManipulation.sh
When I run following command
ps -ef | grep "runDa*"
then sometime I see my process as
username 34535 1 48 11:42:01 - 224:22 /usr/bin/ksh ./runDataManipulation.sh
But if I run exactly same command after couple of seconds then I don't see above process at all.
As my process is running all the time so I expect that ps command to show the process all the time.
If I do grep on the process id of my script like ..
ps -ef | grep 34535
then sometime I see result like
username 34535 1 51 11:42:01 - 229:22 [ksh]
sometime I see result like
username 45678 34535 0 14:12:11 - 0:0 [sed]
My main questions is that ... why do I not see my process when I grep for my process using script name. I am using AIX 6.1.
It looks to me like one script is spawning off your script in another process.
If you look at the results of your ps command below the first line is showing the process id of 34535, this is the main id (say parent id).
username 34535 1 51 11:42:01 - 229:22 [ksh]
This process in turn is firing off another process, this can be seen below, notice the id of the parent process (34535) is mentioned in the line below, the first number is the main process Id and the second number is the calling process.
username 45678 34535
If you changed your ps command to include the sed command you should always see some results if your scripts are still running!
I have a java program that stops often due to errors which is logged in a .log file. What can be a simple shell script to detect a particular text in the last/latest line say
[INFO] Stream closed
and then run the following command
java -jar xyz.jar
This should keep on happening forever(possibly after every two minutes or so) because xyz.jar writes the log file.
The text stream closed can arrive a lot of times in the log file. I just want it to take an action when it comes in the last line.
How about
while [[ true ]];
do
sleep 120
tail -1 logfile | grep -q "[INFO] Stream Closed"
if [[ $? -eq 1 ]]
then
java -jar xyz.jar &
fi
done
There may be condition where the tailed last log "Stream Closed" is not the real last log and the process is still logging the messages. We can avoid this condition by checking if the process is alive or not. If the process exited and the last log is "Stream Closed" then we need to restart the application.
#!/bin/bash
java -jar xyz.jar &
PID=$1
while [ true ]
do
tail -1 logfile | grep -q "Stream Closed" && kill -0 $PID && sleep 20 && continue
java -jar xyz.jar &
PID=$1
done
I would prefer checking whether the corresponding process is still running and restart the program on that event. There might be other errors that cause the process to stop. You can use a cronjob to periodically (like every minute) perform such a check.
Also, you might want to improve your java code so that it does not crash that often (if you have access to the code).
i solved this using a watchdog script that checks directly (grep) if program(s) is(are) running. by calling watchdog every minute (from cron under ubuntu), i basically guarantee (programs and environment are VERY stable) that no program will stay offline for more than 59 seconds.
this script will check a list of programs using the name in an array and see if each one is running, and, in case not, start it.
#!/bin/bash
#
# watchdog
#
# Run as a cron job to keep an eye on what_to_monitor which should always
# be running. Restart what_to_monitor and send notification as needed.
#
# This needs to be run as root or a user that can start system services.
#
# Revisions: 0.1 (20100506), 0.2 (20100507)
# first prog to check
NAME[0]=soc_gt2
# 2nd
NAME[1]=soc_gt0
# 3rd, etc etc
NAME[2]=soc_gp00
# START=/usr/sbin/$NAME
NOTIFY=you#gmail.com
NOTIFYCC=you2#mail.com
GREP=/bin/grep
PS=/bin/ps
NOP=/bin/true
DATE=/bin/date
MAIL=/bin/mail
RM=/bin/rm
for nameTemp in "${NAME[#]}"; do
$PS -ef|$GREP -v grep|$GREP $nameTemp >/dev/null 2>&1
case "$?" in
0)
# It is running in this case so we do nothing.
echo "$nameTemp is RUNNING OK. Relax."
$NOP
;;
1)
echo "$nameTemp is NOT RUNNING. Starting $nameTemp and sending notices."
START=/usr/sbin/$nameTemp
$START 2>&1 >/dev/null &
NOTICE=/tmp/watchdog.txt
echo "$NAME was not running and was started on `$DATE`" > $NOTICE
# $MAIL -n -s "watchdog notice" -c $NOTIFYCC $NOTIFY < $NOTICE
$RM -f $NOTICE
;;
esac
done
exit
i do not use the log verification, though you could easily incorporate that into your own version (just change grep for log check, for example).
if you run it from command line (or putty, if you are remotely connected), you will see what was working and what wasnt. have been using it for months now without a hiccup. just call it whenever you want to see what's working (regardless of it running under cron).
you could also place all your critical programs in one folder, do a directory list and check if every file in that folder has a program running under the same name. or read a txt file line by line, with every line correspoding to a program that is supposed to be running. etcetcetc
A good way is to use the awk command:
tail -f somelog.log | awk '/.*[INFO] Stream Closed.*/ { system("java -jar xyz.jar") }'
This continually monitors the log stream and when the regular expression matches its fires off whatever system command you have set, which is anything you would type into a shell.
If you really wanna be good you can put that line into a .sh file and run that .sh file from a process monitoring daemon like upstart to ensure that it never dies.
Nice and clean =D