Quick bash script for awstats. Noting beautiful, but having an issue with loop not ocurring. Output only shows that it has performed one loop for name1, it never gets to name2, name3 with the printf's.
#!/bin/bash
awstats_command='perl /var/www/html/awstats/wwwroot/cgi-bin/awstats.pl'
html_path="/var/www/html/awstats/wwwroot"
activelogpath="/logs/web/active"
archivelogpath="/logs/web/archive"
day='date +%Y-%m-%d.%H'
# List of web servers we are processing stats for:
for i in name1 name2 name3
do
if [[ $i = "name1" ]]
then
# Custom reports for name1 contains subdirectory statistics
printf "\nProcessing log files for $i...\n"
/usr/bin/perl /var/www/html/awstats/wwwroot/cgi-bin/awstats.pl -config=name1 -update
printf "done.\n"
printf "\nGenerating .html files for $i...\n"
/var/www/html/awstats/wwwroot/cgi-bin/do.reports $i
$awstats_command -config=$i -output=urldetail:/about/ -staticlinks > $html_path/$i/awstats.$i.about.html
printf "done.\n"
else
printf "\nProcessing log files for $i...\n"
# Will do something when working $i
printf "done.\n"
printf "\nGenerating .html files for $i...\n"
# Will do something when working $i
printf "done.\n"
fi
printf "\nCompressing and archiving log files...\n"
exec /usr/bin/gzip -cv "$activelogpath"/"$i"/*.log > "$archivelogpath"/"$i"/"$(date +%Y%m%d_%H%M%S)".gz
# rm -f $activelogpath/$i/*.log
printf "\nCompleted!\n"
done
exec /usr/bin/gzip -cv "$activelogpath"/"$i"/*.log > "$archivelogpath"/"$i"/"$(date +%Y%m%d_%H%M%S)".gz
exec replaces the current process with the named program. Execution does not continue past an exec statement. There's no need for it here. Just call gzip without it.
gzip -cv "$activelogpath/$i"/*.log > "$archivelogpath/$i/$(date +%Y%m%d_%H%M%S).gz"
There's also no need to write /usr/bin/, or to leave and re-enter quotes so often.
Related
I know I can do something like
cat <(cat somefile)
But I want to build up a string of <().
So:
for file in *.file; do
mySubs="${mySubs} <(cat ${file})"
done
cat ${mySubs} #cat <(cat 1.file) <(cat 2.file) ... <(cat something.file)
Without having to use eval.
Use named pipes directly. Use mktemp to create temporary file names for each pipe so that you can remove them after you are done.
fifos=()
for f in file1 file2 file3; do
t=$(mktemp)
mkfifo "$t"
pipes+=("$t")
someCommand "$f" > "$t" &
done
someOtherCommand "${pipes[#]}"
rm "${pipes[#]}"
I'm assuming cat is a standin for a more complicated command. Here, I'm explicitly wrapping it to show that:
#!/usr/bin/env bash
someCommand() { echo "Starting file $1"; cat "$1"; echo "Ending file $1"; }
wrap_all() {
## STAGE 1: Assemble the actual command we want to run
local fd cmd_len retval
local -a cmd fds fd_args
cmd_len=$1; shift
while (( cmd_len > 0 )); do
cmd+=( "$1" )
cmd_len=$((cmd_len - 1))
shift
done
## STAGE 2: Open an instance of someCommand for each remaining argument
local fd; local -a fds
fds=( )
for arg; do
exec {fd}< <(someCommand "$arg")
fds+=( "$fd" )
fd_args+=( "/dev/fd/$fd" )
done
## STAGE 3: Actually run the command
"${cmd[#]}" "${fd_args[#]}"; retval=$?
## STAGE 4: Close all the file descriptors
for fd in "${fds[#]}"; do
exec {fd}>&-
done
return "$retval"
}
Invocation as:
echo "one" >one.txt; echo "two" >two.txt
wrap_all 1 cat one.txt two.txt
...which outputs:
Starting file one.txt
one
Ending file one.txt
Starting file two.txt
two
Ending file two.txt
Note that this requires bash 4.1 for automatic FD allocation support (letting us avoid the need for named pipes).
I've spent some time trying to figure this out with various internet searches and digging through stackoverflow. I'll try to explain this as best as a noob can.
I have a script that searches a config repository directory that is populated with directories for every Juniper and Cisco router and switch we have deployed. Each device directory has a file or two that I'm interested in, "show.version" and "show.chassis.hardware", except when they don't. The second file, "show.chassis.hardware" is not a command that Cisco boxes has, so that file does not exist in Cisco device directories. There also isn't a naming scheme that can easily tell me if the device is Juniper or Cisco, which is also part of the reason why my script exists.
To make things more fun, different models and even software versions output the show.version file in different formats (for both Cisco and Juniper), so my awk is full of all the different fields we will encounter.
Script:
#!/usr/local/bin/zsh
svn="$HOME/svn/nw_config_data/"
case "$1" in
("-a")
hosts=""
;;
("-b")
hosts=".bb.domain.net"
;;
("-c")
hosts=".cpe.domain.net"
;;
("-e")
hosts=".etech.domain.net"
;;
("-k")
hosts=".core.domain.net"
;;
("-m")
hosts=".maint.domain.net"
;;
("-o")
hosts=".ohgov.domain.net"
;;
esac
dirs=($(ls -d $svn*$hosts*))
for hostdir in $dirs
do host=$(echo $hostdir | grep -Eo "(\w|\.|-)*$")
awk -v h=$host '/^Model/{m=$2} /^Model number/{m=$4} /^\*0/{m=$2}
/^JUNOS Base OS boot/{v=$5} /^Junos:/{v="["$2"]"} /^BOOTLDR:/{v=$7}
/^JUNOS EX Software Suite/{v=$5} /^ROM:/{v=$5} /^JUNOS Software
Release/{v=$4} /^Chassis/{s=$2} /^Motherboard serial number/{s=$3}
END {if(m!="number") {printf("%s %s %s %s\n",h,m,v,s)}}'
"$hostdir/show.version" "$hostdir/show.chassis.hardware"
done
What it looks like when I run the script:
% cver -b
device1-e0.bb.domain.net ex4300-24t [14.1X53-D25.2] Serial#
device2-e0.bb.domain.net ex4300-24t [14.1X53-D25.2] Serial#
awk: can't open file /home/clmbn eng2/a/rwalker/svn/nw_config_data/device3-e1.bb.domain.net/show.chassis.hardware
input record number 55, file /home/clmbn-eng2/a/rwalker/svn/nw_config_data/device3-e1.bb.domain.net/show.chassis.hardware
source line number 1
device4-r0.bb.domain.net m7i [13.3R6.5] Serial#
...
What I want it to look like
% cver -b
device1-e0.bb.domain.net ex4300-24t [14.1X53-D25.2] Serial#
device2-e0.bb.domain.net ex4300-24t [14.1X53-D25.2] Serial#
device3-e1.bb.domain.net C3750 12.1(14r)EA1a, Serial#
device4-r0.bb.domain.net m7i [13.3R6.5] Serial#
...
I have 13 directories that do not have the "show.chassis.hardware" file but do have the "show.version" file which does have all the information I need from it. I have one directory that has no files, but it doesn't matter because that device is going to be replaced.
From what I've been reading, awk might not be able to do this, but I have faith that someone out there knows a way to make it work. If my approach (shell & awk scripting) just isn't going to work and I need to do it in something else (Perl or Python for example) I'll be completely stuck until I can learn those enough to convert my script to one of those languages.
Also, we don't have bash installed on this server and I don't know when we will since I'm not the admin.
You need
-f file
true if file exists and is a regular file.
if [[ -f "$hostdir/show.version" && -f "$hostdir/show.chassis.hardware" ]]; then
# your awk command goes here...
awk '{ }' "$hostdir/show.version" "$hostdir/show.chassis.hardware"
else
echo "not enough files found"
fi
You may refer : http://zsh.sourceforge.net/Doc/Release/Conditional-Expressions.html
--edit--
That's cool that this bit of script skips the directories that doesn't
have the file, but it doesn't pull the information from the
"show.version" file and print that information anyway. So the output
shows device1, device2, device 4...
Here is code snippet
function myfunc(){
# replace with your awk
awk '{ print }' "$#"
}
if [[ -f "$hostdir/show.version" && -f "$hostdir/show.chassis.hardware" ]]; then
# your awk command goes here...
myfunc "$hostdir/show.version" "$hostdir/show.chassis.hardware"
else
echo "not enough files found"
# pass only one file, version file
myfunc "$hostdir/show.version"
fi
You can use zsh's (...|...) globbing:
$ mkdir foo bar; touch {foo,bar}/show.version foo/show.chassis.hardware
$ echo foo/show.(version|chassis.hardware)
foo/show.chassis.hardware foo/show.version
$ echo bar/show.(version|chassis.hardware)
bar/show.version
Since this is globbing, it will only expand to existing files. So your awk command would look like:
awk -v h=$host '/^Model/{m=$2} ... END {...}' "$hostdir"/show.(version|chassis.hardware)
(omitting the awk script for readability)
I'd also simplify your script a bit using associative arrays instead of cases:
#!/usr/local/bin/zsh
usage () {
echo "Help!" # echo your help message here
exit $1
}
svn="$HOME/svn/nw_config_data/"
declare -A hosts # make associative array
hosts["-a"]=""
hosts["-b"]=".bb.domain.net"
hosts["-c"]=".cpe.domain.net"
hosts["-e"]=".etech.domain.net"
hosts["-k"]=".core.domain.net"
hosts["-m"]=".maint.domain.net"
hosts["-o"]=".ohgov.domain.net"
if [[ $1 == -h ]]
then
usage
elif (( ${+hosts["$1"]} )) # check if $1 is a key in hosts
then
echo "Invalid option: $1"
usage 1 # exit with status 1 to indicate error
fi
dirs=( $svn*$hosts["$1"]* ) # no need for ls here
for hostdir in $dirs
do
host=$(echo $hostdir | grep -Eo "(\w|\.|-)*$")
awk -v h=$host '
/^Model/{m=$2}
/^Model number/{m=$4}
/^\*0/{m=$2}
/^JUNOS Base OS boot/{v=$5}
/^Junos:/{v="["$2"]"}
/^BOOTLDR:/{v=$7}
/^JUNOS EX Software Suite/{v=$5}
/^ROM:/{v=$5}
/^JUNOS Software Release/{v=$4}
/^Chassis/{s=$2}
/^Motherboard serial number/{s=$3}
END {if(m!="number") {printf("%s %s %s %s\n",h,m,v,s)}}' \
"$hostdir"/show.(version|chassis.hardware)
done
Alternately, you can use a concise case with the array:
declare -A hosts # make associative array
hosts["-a"]=""
hosts["-b"]=".bb.domain.net"
hosts["-c"]=".cpe.domain.net"
hosts["-e"]=".etech.domain.net"
hosts["-k"]=".core.domain.net"
hosts["-m"]=".maint.domain.net"
hosts["-o"]=".ohgov.domain.net"
case $1 in
-[abcekmno]) dirs=( $svn*${hosts["$1"]}* )
;;
-h|help) usage
;;
*) echo "Invalid option: $1"
usage 1 # exit with status 1 to indicate error
;;
esac
Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)
title: bash parameter expansion within a scalar variable
I have a bash script which runs a diff between two files.
If there is a diff, I want it to print statement1 and statement2
They are long so i put them into variables, but the echo statement
will not expand the parameter.
Can this be done in bash?
#!/bin/bash
set -x
source="/home/casper"
target="data/scripts"
statement1="There is a change in ${i}, please check the file"
statement2="or cp /home/casper/${i} /data/scripts/$i"
for i in file1 file2l file3 file4 file5 ; do
sleep 1 ;
if diff $source/$i $target/$i 2>&1 > /dev/null ; then
echo " "
else
echo "$statement1 "
echo "$statement2 "
fi
done
exit 0
The script seems to work - it finds a diff when it needs to find one.
However this is what it prints out.
There is a change in , please check the file
or cp /home/casper/ data/scripts/
I want it to say
There is a change in file2, please check the file
or cp /home/casper/file2 /data/scripts/file2
The problem is that $i is expanded when you define statement1 and statement2, not when you expand them. Use a shell function to output the text.
notification () {
echo "There is a change in $1, please check the file"
echo "or cp /home/casper/$1 /data/scripts/$1"
}
source="/home/casper"
target="data/scripts"
for i in file1 file2l file3 file4 file5 ; do
sleep 1 ;
if diff "$source/$i" "$target/$i" 2>&1 > /dev/null ; then
echo " "
else
notification "$i"
fi
done
exit 0
This can be done using eval:
TEMPLATE_MSG="aaa \${VALUE} ccc"
...
VALUE="bbb"
eval echo "${TEMPLATE_MSG}"
But I don't recommend it, because eval is evil :-) Other option is using pattern substitution:
TEMPLATE_MSG="aaa #1# ccc"
...
VALUE="bbb"
echo "${TEMPLATE_MSG/#1#/${VALUE}}"
So you put some unique pattern in your message (e.g. #1#) and then, when you print the message, you replace it with the content of variable.
Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)