Anyone know a better way to do this were it is faster? This currently is slow when pushing high lines per second to this script:
#!/bin/bash
declare -A clientarray
file=$1
timer=$2
e=$(date --date "now +$timer second" +%s)
while read line
do
if [ -n "${clientarray[$line]}" ]; then
let "clientarray[$line]=clientarray[$line]+1"
echo "$line: ${clientarray[$line]}"
elif [ -z "${clientarray[$line]}" ]; then
clientarray[$line]=1
echo "$line: ${clientarray[$line]}"
fi
if [ $(date +%s) -gt $e ]; then
e=$(date --date "now +$timer second" +%s)
fi
done < <(tail -F $file | gawk -F"]" '/]/ {print $1}')
Here is an example of the lines:
someline]
someline2]
somethingidontwant
someline3]
somethingelseidontwant
someline4]
and to call the script:
bash script.sh somelogfile.log 1
If I comment out the if logic at the very end it goes really fast but with it the speed drops 2/3rds. Tested it with pv:
(this is with the if logic):
ubuntu#myhost:~/graphs$ tail -F somelogfile.log | pv -N RAW -lc >/dev/null |
> bash script.sh somelogfile.log 1 | pv -N SCP -lc >/dev/null
RAW: 2.18k 0:00:16 [ 493/s ] [ <=> ]
SCP: 593 0:00:16 [ 150/s ] [ <=> ]
(this is without)
ubuntu#myhost:~/graphs$ tail -F somelogfile.log | pv -N RAW -lc >/dev/null |
> bash script.sh somelogfile.log 1 | pv -N SCP -lc >/dev/null
RAW: 7.69k 0:00:15 [512/s] [ <=> ]
SCP: 7.6k 0:00:15 [503/s] [ <=> ]
Let me know if I am missing something on my script or testing side, especially any "DOH!"'s.
I think at this point I would love one =)
As a guess, I'd say it could be that that last if...fi block adds two non-builtin commands per iteration. Everything else in the loop is bash builtins, which execute much faster. With it, you have a call to date within the test, and another in the body of the if. In addition, date --date has to parse and evaluate that "now +$timer second" expression each time it's called, which probably isn't very speedy, given --date's generality. If I were you, I'd try reimplementing this in a scripting language with more native handling of dates/times: Perl, Ruby, Python, whatever you're comfortable with.
You also appear to have a bug:
if [ `date +%s` > $e ] ...
This says: execute the command date +%s and interpolate its output (say 12345) into another command [ 12345 > $e ] (so far so good). That command says: run the [ builtin with two arguments (12345 and ]), and redirect its standard output stream to a file named by the value of $e (uh-oh). You probably want to use -gt instead of > here.
I'm not sure what you are doing with $e, but you can print the current date using the shell builtin printf much faster than you can by calling out to date. Subprocess calls tend to be expensive. For example, if you are not on glibc2 you can do:
printf '%(%+)T\n' -1
to get exactly the output of the date command. %+ is not supported on glibc2 so you can construct something identical with other parameters, or something similar with:
printf '%(%c %Z)T\n' -1
If you need to capture and process the date somehow then you may still need a subshell call using $() but there's a decent chance it's still faster than date.
Related
I'm trying to run a ash script that constantly checks how many characters are in a file and execute some code if it reaches at least 170 characters or if 5 seconds have passed. To do that I wanted to call wc -c, however it keeps telling me it has an unknown operand.
The code:
#!/bin/ash
while true; do
secs=5
endTime=$(( $(date +%s) + secs ))
while [ /usr/bin/wc -c < "/tmp/regfile_2" -gt 170 ] || [ $(date +%s) -lt $endTime ]; do
#more code
It's output is ash: -c: unknown operand
You want to check whether the output from wc meets a specific condition. To do that, you need to actually execute wc, just like you already do with date to examine its output.
while [ $(wc -c < "/tmp/regfile_2") -gt 170 ] ||
[ $(date +%s) -lt $endTime ]; do
# ...stuff
Notice the $(command substitution) around the wc command.
As you can see from the answer to the proposed duplicate Checking the success of a command in a bash `if [ .. ]` statement your current command basically checks whether the static string /usr/bin/wc is non-empty; the -c after this string is indeed unexpected, and invalid syntax.
(It is unclear why you hardcode the path to wc; probably just make sure your PATH is correct before running this script. There are situations where you do want to hardcode paths, but I'm guessing this isn't one of them; if it is, you should probably hardcode the path to date, too.)
I have never used bash before but I am trying to understand this piece of code. The script is supposed to display all log in names, full names and their user-ids. However, whenever I run I can not get past the first if statement and if I delete the statement, it does not work.
#!/bin/bash
if [ $# -lt 1 ];
then
printf "Not enough arguments - %d\n" $#
exit 0
fi
typeset user=""
typeset name=""
typeset passwdEntry=""
while [ $# -ge 1 ];
do
user=$1
shift
name=""
passwdEntry=`grep -e ^$user /etc/passwd 2>/dev/null`
if [ $? -eq 0 ]; then
name=`echo $passwdEntry|awk -F ':' '{print $5}'`
fi
echo "$user $name"
done
$# means "the number of arguments to the current Bash program", and $1 means "the first argument to the current Bash program".
So your problem is that you're not passing any arguments to the program; for example, instead of something like this:
./foo.sh
you'll need to write something like this:
./foo.sh USERNAME
As you are new to Bash, I highly recommend skimming and bookmarking the Bash Reference Manual, http://www.gnu.org/software/bash/manual/bashref.html. It's all on a single page, so you can use your browser's "find in page" function (typically Ctrl+F) to search for things.
I am trying to create a "Lambda" style WHERE script.
I want lambdaWHERE to take piped input and pass it through if condition after given as arguments is met. Like xargs I use {} to represent what comes down the pipe.
I call command like:
ls -d EqAAL* | lambdaWHERE.sh -f {}/INFO_ACTIVETICK
I want the folder names passed through if they contain a file called INFO_ACTIVETICK
Here is the script:
#!/bin/sh
#set -x
ARGS=$*
while read i
do
CMD=`echo $ARGS | sed 's/{}/'$i'/g'`
if [[ $CMD ]]
then
echo $i
fi
done
But when I run it a mysterious "-n" appears...
$ ls -d EqAAL* | /q/lambdaWHERE.sh -f {}/INFO_ACTIVETICK
+ ARGS='-f {}/INFO_ACTIVETICK'
+ read i
++ echo -f '{}/INFO_ACTIVETICK'
++ sed 's/{}/EqAAL-1m/g'
+ CMD='-f EqAAL-1m/INFO_ACTIVETICK'
+ [[ -n -f EqAAL-1m/INFO_ACTIVETICK ]]
+ echo EqAAL-1m
EqAAL-1m
+ read i
How can I get the bit in the [[ ]] correct?
You were quite close. you only need to switch to the standard POSIX [ $CMD ] and it will work.
The main difference between using [[ $CMD ]] and [ $CMD ] is that the first has fewer surprises and you need not quote variables. That also means that a variable is though of as one token and cannot have a whole expression in it like you are trying. [ $CMD ] however works the same way as the original shell where [ was just a command an thus need explicit quotations in order to interpret something with spaces as one argument.
There is a relevant question about the differences between [[ ...]] and [ ..]
I am new to writing scripts and am trying to start out with a simple one. I am stumped as to why I am receiving the error: [: 13: Illegal number: count from the code below. Line 13 is the last fi.
count=grep "^$(date -d -30minute +'%Y-%m-%d %H:%M')" /var/log/****/zlsapp.log | wc -l
if [ count -ge 50 ]
then
if [ count -lt 100 ]
then
exit 1
fi
if [ count -ge 100 ]
then
exit 2
fi
exit 0
fi
Also is there anyway to do compound if statements like if(count >= 50 && count < 100)?
Two reasons. (1) in bash variables are untyped (could be int, could be char). In order to remove ambiguity, you can specify the type with:
declare -i count
To tell bash it should be an int. (2) you need to dereference your variables with $ to get the number back. I.E.
[ $count -lt 100 ]
(it is also good practice to quote your variables - not required, but good practice: [ "$count" -lt 100 ]. Drop a comment if you still have trouble.
The line:
count=grep "^$(date -d -30minute +'%Y-%m-%d %H:%M')" /var/log/zumigo/zlsapp.log | wc -l
probably does not do what you think it does (or you've not accurately copied and pasted your actual code into the question). If run in the middle of 2014-08-01, it runs the command "2014-08-01 12:00" with the log file as an argument and the environment variable count set to the value grep, and pipes the output from the probably non-existent command to wc -l.
When you subsequently go to test $count in the test statements, it is an empty string, which doesn't convert properly to a number.
You probably meant:
count=$(grep "^$(date -d -30minute +'%Y-%m-%d %H:%M')" /var/log/zumigo/zlsapp.log | wc -l)
This captures the output of running grep on the log file and piping the output to wc -l.
If you run your script with bash -x or equivalent (the -x option usually shows the execution of the script), you should see this.
The problem is that count does not refer to the variable count; it's simply the string "count".
Change:
if [ count -ge 50 ]
to
if [ $count -ge 50 ]
and make the corresponding change elsewhere (but not in the initial assignment).
You should also use double quotes:
if [ "$count" -ge 50 ]
Also is there anyway to do compound if statements like if(count >= 50 && count < 100)?
Yes:
if [ "$count" -ge 50 -a "$count" -lt 100 ]
is likely to work, but the -a (logical and) operator is marked as obsolescent by POSIX. Instead write
if [ "$count" -ge 50 ] && [ "$count" -lt 100 ]
If you're using bash, info bash and search for the "test" command ([ is an alias for test).
And if you're using bash, you should consider using [[ ... ]] rather than [ ... ] -- or you can use (( ... )) for arithmetic expressions. See the bash documentation for more information (follow the iink or type info bash).
In addition to the missing $ signs, the first line of your script:
count=grep "^$(date -d -30minute +'%Y-%m-%d %H:%M')" /var/log/****/zlsapp.log | wc -l
doesn't set the count variable, since you're not capturing the output of the grep ... | wc -l command. To do so:
count=$(grep "^$(date -d -30minute +'%Y-%m-%d %H:%M')" /var/log/****/zlsapp.log | wc -l)
(Yes, $(...) can be nested.)
I want to write a function for when I have something like the following
echo 1 2 3|pick
Pick will then take the arguments and I will do something with them.
How do I do this?
Are you looking for xargs?
pick() {
read -r arg1 arg2 remainder
echo first arg is $arg1
echo The remaining args are $remainder
}
--EDIT (response to question in comment)
One way to loop through the arguments:
pick() {
read args;
set $args;
while test $# -ne 0; do
echo $1
shift
done
}
On each iteration of the loop, $1 refers to an argument.
If I'm not mistaken, the OP wants the same thing I do: you feed it a string, and if the string containes multiple {words,lines}, it presents you a menu, and you pick one, and it returns the one you pick on stdout.
If there's only one item, it just returns it.
This is useful for--to use my particular use-case--a log file viewer script: you give it a substring of a filename, and it greps through find /var/log -name \*$arg\* -print to see what it can find. If it gets a unique hit, it hands it back to your script, which runs less against it. If it gets more than one hit, it shows you a menu, and lets you pick one.
ISTR that KSH has a builtin for this, but that I wasn't all that impressed with it; I don't recall if bash has one.
I am here because I was searching to see if someone had already written it before writing it myself. :-)
UPDATE: Nope; I wrote it myself:
Here's some example code:
/usr/local/bin/msg:
PATH=$PATH:/usr/local/bin
[ $UID = 0 ] || exec sudo su root -c "$0 $*"
FILE=/var/log/messages
[ $# -eq 1 ] &&
FILE=`find /var/log/ -name \*$1\* -print |
egrep -v '2011|.[0-9]$' |
pick`
echo "$FILE"
less +F $FILE
Since I'm piping the name to less +F I want to grep out archived log files; this is for interactive log viewing.
/usr/local/bin/pick:
# Present the user a bash Select menu, and let them pick
# Try to be smart about multi-line responses
# must take input on stdin if it might be multiline
# get multiline input from stdin
while read LINE </dev/stdin
do
CHOICES+=( $LINE )
done
# add on anything specified as arguments
while [ $# -gt 0 ]
do
CHOICES+=( $1 )
shift
done
# if only one thing to pick, just pick it
if [ ${#CHOICES[*]} -eq 1 ]
then
echo $CHOICES
exit
fi
# eval set $CHOICES
select CHOSEN in ${CHOICES[#]}
do
echo $CHOSEN
exit
done </dev/tty