I'm writing a routine that will identify if a process stops running and will do something once the processes targeted is gone.
I came up with this code (as a test for my future code):
#!/bin/bash
value="aaa"
ls | grep $value
while [ $? = 0 ];
do
sleep 5
ls | grep $value
echo $?
done;
echo DONE
My problem is that for some reason, the loop never stops and echoes 1 after I delete the file "aaa".
0
0 >>> I delete the file at that point (in another terminal)
1
1
1
1
.
.
.
I would expect the output to be "DONE" as soon as I delete the file...
What's the problem?
SOLUTION:
#!/bin/bash
value="aaa"
ls | grep $value
while [ $? = 0 ];
do
sleep 5
ls | grep $value
done;
echo DONE
The value of $? changes very easily. In the current version of your code, this line:
echo $?
prints the status of the previous command (grep) -- but then it sets $? to 0, the status of the echo command.
Save the value of $? in another variable, one that won't be clobbered next time you execute a command:
#!/bin/bash
value="aaa"
ls | grep $value
status=$?
while [ $status = 0 ];
do
sleep 5
ls | grep $value
status=$?
echo $status
done;
echo DONE
If the ls | grep aaa is intended to check whether a file named aaa exists, this:
while [ ! -f aaa ] ; ...
is a cleaner way to do it.
$? is the return code of the last command, in this case your sleep.
You can rewrite that loop in much simpler way like this:
while [ -f aaa ]; do
sleep 5;
echo "sleeping...";
done
You ought not duplicate the command to be tested. You can always write:
while cmd; do ...; done
instead of
cmd
while [ $? = 0 ]; do ...; cmd; done
In your case, you mention in a comment that the command you are testing is parsing the output of ps. Although there are very good arguments that you ought not do that, and that the followon processing should be done by the parent of the command for which you are waiting, we'll ignore that issue at the moment. You can simply write:
while ps -ef | grep -v "grep mysqldump" |
grep mysqldump > /dev/null; do sleep 1200; done
Note that I changed the order of your pipe, since grep -v will return true if it
matches anything. In this case, I think it is not necessary, but I believe is more
readable. I've also discarded the output to clean things up a bit.
Presumably your objective is to wait until a filename containing the string $value is present in the local directory and not necessarily a single filename.
try:
#!/bin/bash
value="aaa"
while ! ls *$value*; do
sleep 5
done
echo DONE
Your original code failed because $?is filled with the return code of the echo command upon every iteration following the first.
BTW, if you intend to use ps instead of ls in the future, you will pick up your own grep unless you are clever. Use ps -ef | grep [m]ysqlplus.
Related
Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)
I am looking to create a shell script to read the message log and when finds the correct string perform an action. So far I have the following:
#!/bin/bash
string="ntp engine ready"
tail -n 0 -f /var/log/messages | \
while read LINE
do
echo "$LINE | grep -q $string"
if [ $? == 0];then
shttpclient "http://127.0.0.1/do/action"
fi
done
But, I get the following error:
grep: engine: No such file or directory
grep: ready: No such file or directory
Even when I see the logger has outputted ntp engine ready.
Firstly, you need to fix your quotes:
echo "$LINE" | grep -q "$string"
Secondly, you can simply do:
if echo "$LINE" | grep -q "$string"; then
rather than checking the return code $? manually. Remember that [ is a command too and if is just checking its return code.
If you do need to use [, remember that ] is an argument to the command so it is essential to surround it with spaces:
if [ $? = 0 ]
I have also removed the second = as it is a bash extension to support it. Actually you are doing an integer comparison, so really it should be one of the following:
if [ $? -eq 0 ] # POSIX compliant
if (( $? == 0 )) # bash arithmetic context
Alter the line as follows:
echo "$LINE" | grep -q "$string"
The quotes were not set correctly. Like when you execute that: grep -q ntp engine ready; ntp is the string to search and engine and ready are the files. It must look like: grep -q "ntp engine ready".
I am writing a script that will take in 3 outputs and then search all files within a predefined path. However, my grep command seems to be breaking the script with error code 123. I have been staring at it for a while and cannot really seem the error so I was hoping someone could point out my error. Here is the code:
#! /bin/bash -e
#Check if path exists
if [ -z $ARCHIVE ]; then
echo "ARCHIVE NOT SET, PLEASE SET TO PROCEED."
echo "EXITING...."
exit 1
elif [ $# -ne 3 ]; then
echo "Illegal number of arguments"
echo "Please enter the date in yyyy mm dd"
echo "EXITING..."
exit 1
fi
filename=output.txt
#Simple signal handler
signal_handler()
{
echo ""
echo "Process killed or interrupted"
echo "Cleaning up files..."
rm -f out
echo "Finsihed"
exit 1
}
trap 'signal_handler' KILL
trap 'signal_handler' TERM
trap 'signal_handler' INT
echo "line 32"
echo $1 $2 $3
#Search for the TimeStamp field and replace the / and : characters
find $ARCHIVE | xargs grep -l "TimeStamp: $2/$3/$1"
echo "line 35"
fileSize=`wc -c out.txt | cut -f 1 -d ' '`
echo $fileSize
if [ $fileSize -ge 1 ]; then
echo "no"
xargs -n1 basename < $filename
else
echo "NO FILES EXIST"
fi
I added the echo's to know where it was breaking. My program prints out line 32 and the args but never line 35. When I check the exit code I get 123.
Thanks!
Notes:
ARCHIVE is set to a test directory, i.e. /home/'uname'/testDir
$1 $2 $3 == yyyy mm dd (ie a date)
In testDir there are N number of directories. Inside these directories there are data files that have contain data as well as a time tag. The time tag is of the following format: TimeStamp: 02/02/2004 at 20:38:01
The scripts goal is to find all files that have the date tag you are searching for.
Here's a simpler test case that demonstrates your problem:
#!/bin/bash -e
echo "This prints"
true | xargs false
echo "This does not"
The snippet exits with code 123.
The problem is that xargs exits with code 123 if any command fails. When xargs exits with non-zero status, -e causes the script to exit.
The quickest fix is to use || true to effectively ignore xargs' status:
#!/bin/bash -e
echo "This prints"
true | xargs false || true
echo "This now prints too"
The better fix is to not rely on -e, since this option is misleading and unpredictable.
xargs makes the error code 123 when grep returns a nonzero code even just once. Since you're using -e (#!/bin/bash -e), bash would exit the script when one of its commands return a nonzero exit code. Not using -e would allow your code to continue. Just disabling it on that part can be a solution too:
set +e ## Disable
find "$ARCHIVE" | xargs grep -l "TimeStamp: $2/$1/$3" ## If one of the files doesn't match the pattern, `grep` would return a nonzero code.
set -e ## Enable again.
Consider placing your variables around quotes to prevent word splitting as well like "$ARCHIVE".
-d '\n' may also be required if one of your files' filename contain spaces.
find "$ARCHIVE" | xargs -d '\n' grep -l "TimeStamp: $2/$1/$3"
i'm working on a small bash script which counts how often a script with a certain name is running.
ps -ef | grep -v grep | grep scrape_data.php | wc -l
is the code i use, via ssh it outputs the number of times scrape_data.php is running. Currently the output is 3 for example. So this works fine.
Now I'm trying to make a little script which does something when the count is smaller than 1.
#!/bin/sh
if [ ps -ef | grep -v grep | grep scrape_data.php | wc -l ] -lt 1; then
exit 0
#HERE PUT CODE TO START NEW PROCESS
else
exit 0
fi
The script above is what I have so far, but it does not work. I'm getting this error:
[root#s1 crons]# ./check_data.sh
./check_data.sh: line 4: [: missing `]'
wc: invalid option -- e
What am I doing wrong in the if statement?
Your test syntax is not correct, the lt should be within the test bracket:
if [ $(ps -ef | grep -v grep | grep scrape_data.php | wc -l) -lt 1 ]; then
echo launch
else
echo no launch
exit 0
fi
or you can test the return value of pgrep:
pgrep scrape_data.php &> /dev/null
if [ $? ]; then
echo no launch
fi
if you're using Bash then drop [ and -lt and use (( for arithmetic comparisons.
ps provides the -C switch, which accepts the process name to look for.
grep -v trickery are just hacks.
#!/usr/bin/env bash
proc="scrape_data.php"
limit=1
numproc="$(ps hf -opid,cmd -C "$proc" | awk '$2 !~ /^[|\\]/ { ++n } END { print n }')"
if (( numproc < limit ))
then
# code when less than 'limit' processes run
printf "running processes: '%d' less than limit: '%d'.\n" "$numproc" "$limit"
else
# code when more than 'limit' processes run
printf "running processes: '%d' more than limit: '%d'.\n" "$numproc" "$limit"
fi
Counting the lines is not needed. Just check the return value of grep:
if ! ps -ef | grep -q '[s]crape_data.php' ; then
...
fi
The [s] trick avoids the grep -v grep.
While the top-voted answer does in fact work, I have a solution that I used for my scraper that worked for me.
<?php
/**
* Go_Get.php
* -----------------------------------------
* #author Thomas Kroll
* #copyright Creative Commons share alike.
*
* #synopsis:
* This is the main script that calls the grabber.php
* script that actually handles the scraping of
* the RSI website for potential members
*
* #usage: php go_get.php
**/
ini_set('max_execution_time', 300); //300 seconds = 5 minutes
// script execution timing
$start = microtime(true);
// how many scrapers to run
$iter = 100;
/**
* workload.txt -- next record to start with
* workload-end.txt -- where to stop at/after
**/
$s=(float)file_get_contents('./workload.txt');
$e=(float)file_get_contents('./workload-end.txt');
// if $s >= $e exit script otherwise continue
echo ($s>=$e)?exit("Work is done...exiting".PHP_EOL):("Work is not yet done...continuing".PHP_EOL);
echo ("Starting Grabbers: ".PHP_EOL);
$j=0; //gotta start somewhere LOL
while($j<$iter)
{
$j++;
echo ($j %20!= 0?$j." ":$j.PHP_EOL);
// start actual scraping script--output to null
// each 'grabber' goes and gets 36 iterations (0-9/a-z)
exec('bash -c "exec nohup setsid php grabber.php '.$s.' > /dev/null 2>&1 &"');
// increment the workload counter by 36 characters
$s+=36;
}
echo PHP_EOL;
$end = microtime(true);
$total = $end - $start;
print "Script Execution Time: ".$total.PHP_EOL;
file_put_contents('./workload.txt',$s);
// don't exit script just yet...
echo "Waiting for processes to stop...";
// get number of php scrapers running
exec ("pgrep 'php'",$pids);
echo "Current number of processes:".PHP_EOL;
// loop while num of pids is greater than 10
// if less than 10, go ahead and respawn self
// and then exit.
while(count($pids)>10)
{
sleep(2);
unset($pids);
$pids=array();
exec("pgrep 'php'",$pids);
echo (count($pids) %15 !=0 ?count($pids)." ":count($pids).PHP_EOL);
}
//execute self before exiting
exec('bash -c "exec nohup setsid php go_get.php >/dev/null 2>&1 &"');
exit();
?>
Now while this seems like a bit of overkill, I was already using PHP to scrape the data (like your php script in the OP), so why not use PHP as the control script?
Basically, you would call the script like this:
php go_get.php
and then just wait for the first iteration of the script to finish. After that, it runs in the background, which you can see if you use your pid counting from the command line, or a similar tool like htop.
It's not glamorous, but it works. :)
Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)