Handling logs of bash script and comments in text file - bash

I am trying to read a text file which has few commented starts with '#', my bash script should read the lines of the text file which doesn't start with '#'.
Also im trying to capture the output of echo statements in both logs and to show it console window for the user understanding.
I have tried to use the below query for capturing logs and printing in console
exec 2>&1 1>>$logfile
For reading each line of the file and calling the function, i have declared an array and to eliminate lines which starts with '#' , i have used the below query.
declare -a cmd_array
while read -r -a cmd_array | grep -vE '^(\s*$|#)'
do
"${cmd_array[#]}"
done < "$text_file"
Note : I need to eliminate the line starts with '#' and remaining lines to be read and place in array as declared.
Bash script
***********
#! /bin/bash
Function_1()
{
now=$( date '+%Y%m%d%H%M' )
eval logfile="$1"_"$now".log
exec 2>&1 1>>$logfile ### Capture echo output in log and printing in console
#exec 3>&1 1>>$logfile 2>&1
echo " "
echo "############################"
echo "Function execution Begins"
echo "############################"
echo "Log file got created with file name as $1.log"
eval number=$1
eval path=$2
echo "number= $number"
ls -lR $path >> temp.txt
if [ $? -eq 0 ]; then
echo " Above query executed."
else
echo "Query execution failed"
fi
echo "############################"
echo "Function execution Ends"
echo "############################"
echo " "
}
text_file=$1
echo $text_file
declare -a cmd_array ### declaring a array
while read -r -a cmd_array | grep -vE '^(\s*$|#)' ### Read each line in the file with doesnt starts with '#' & keep it in array
do
"${cmd_array[#]}"
done < "$text_file"
Text file
*********
####################################
#Test
#Line2
####################################
Function_1 '125' '' ''
Function_1 '123' '' ''

Consider piping the grep output into the read:
declare -a cmd_array ### declaring a array
### Read each line in the file with doesnt starts with '#' & keep it in array
grep -vE '^(\s*$|#)' < "$text_file" | while read -r -a cmd_array
do
"${cmd_array[#]}"
done
I'm not clear about the output/logging comment. If you need the output appended to a file, in addition to stdout/console), consider using the 'tee' (probably 'tee -a')

I tested with the input file inputfile
echo a
Function_1 '125' '' ''
# skip me
Function_1 '123' '' ''
echo b
and wrote this script:
declare -a cmd_array ### declaring a array
while read -r -a cmd_array
do
echo "${cmd_array[#]}"
"${cmd_array[#]}"
echo
done < <(grep -vE '^(\s*$|#)' inputfile)
For showing output in log and console, see https://unix.stackexchange.com/a/145654/57293
As #GordonDavisson suggested in a comment, you get a simular result with
source inputfile
ignoring comments and empty lines, and calling functions, so I am not sure why you would want an array. This command can be included in your master script, you do not need to modify the inputfile.
Another advantage of sourcing the input is the handling of multi-line input and # in strings:
Function_1 '123' 'this is the second parameter, the third will be on the next line' \
'third parameter for the Function_1 call'
echo "This echo continues
on the next line."
echo "Don't delete # comments in a string"
Function_1 '124' 'Parameter with #, interesting!' ''

Related

Refactoring my shell scipts : calling a script and redirect output to a new file inside a loop

I have a shell script that uses the sed command to transform an input file to a new output file, this script works when I call :
./sed_command.sh input_file > output_file
here is my script :
#!/usr/bin/env sh
while read line
do
# Split each line based on ; separator
transposed=$(echo "$line" | sed -e "s/;/\\n/g")
res=$(echo "$transposed" | sed '/${GDATEF(\([^,]*\),ddMMyyyy)}/{
# Split the string in two part
s//\1/g
# # parse the input for GNU date input format
s/D/ day/g
# Handle shell quoting
'"s/'/\\''\\'/"'
# pass and execute date
s/.*/date -d "now "'\''&'\'' +%d%m%Y/e
}')
oneline=$(echo "$res" | sed -z "s/\n/;/g" | sed 's/.$//')
echo $oneline
done < $1
Now I need to call that script inside a double loop that browse all directories in a folder :
cd /etc/newman/Newman
for team in *;do
if [ -d "$team" ]; then
echo "team=$team"
cd $team
for scope in *;do
if [ -d "$scope" ]; then
echo "scope=$scope"
CSV_FILE=$(ls /etc/newman/Newman/$team/$scope/*.csv -1 || true)
echo "Transforming : ${CSV_FILE}"
if [ ! -z "${CSV_FILE}" ] ; then sh sed_command.sh ${CSV_FILE} > ${CSV_FILE}_Newman; else echo "No CSV files to transform"; fi
fi
done
cd ..
fi
done
I have tested the second script looping works fine the only issue is when I call sed_command.sh inside the second script I get theses two errors :
: not found.sh: line 2:
sed_command.sh: line 19: syntax error: unexpected "done" (expecting "do")
I guess redirection does not work the same way inside a loop? thanks for help

Variable scope in Bash [duplicate]

Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)

How to terminate a cat pipe command in shell script?

I use a command like to cat a pipe file and grep some data. A simple code such as,
temp=""
temp=$(cat file|grep "some data"| wc -c)
if [ $temp -gt 0 ]
then
echo "I got data"
fi
The file is a pipe(FIFO), it will output data and not stop. How can i to terminate the command of cat pipe in a finite time?
grep|wc is the wrong tool for this job. Choose a better one, such as sed,
if sed -n -e '/some data/q;$q1' file; then
....
fi
awk,
found=$(awk '/some data/{print"y";exit}' file)
if [ -n "$found" ]; then
....
fi
or sh itself.
found=
while read line; do
if expr "$line" : ".*some data" >/dev/null; then
found=y
break
fi
done <file
if [ -n "$found" ]; then
....
fi
I got it adding $ to temp variable in line 3:
if [ $temp -gt 0 ]
Because you want to compare temp value, and you get it using $ before the variable.
About file "pipe", you can execute cat until you get a specific string.
I mean, you can use cat for reading and stop when you receive, for example, a "\n".
I will give you an example that you can run in your terminal:
cat > example_file.txt << EOF
hello
I'm a example filen
EOF
cat will be reading from standard input untill you enter "EOF". And then, the content of the file will be:
cat example_file.txt
hello
I'm an example file
So this way you can read by chunks, for example, lines.
Just check the exit status of grep itself:
if grep -q "some data" file; then
echo "I got data"
fi
The -q prevents anything from being written to standard output if a match is found.
Another way to do it is by using shell script.
cat <some file and conditions> &
< perform your task>
kill $(pidof cat)
This works as long as you have one instance of "cat" running at a time.
You can use timeout command, which is part of coreutils.
man timeout:
NAME
timeout - run a command with a time limit
SYNOPSIS
timeout [OPTION] DURATION COMMAND [ARG]...
...
To wait 10 seconds:
temp=$(timeout 10 cat file|grep "some data"| wc -c)
if [ $temp -gt 0 ]
then
echo "I got data"
fi

Test whether stdout has been written to

I have a script that prints in a loop. I want the loop to print differently the first time from all other times (i.e., it should print differently if anything has been printed at all). I am thinking a simple way would be to check whether anything has been printed yet (i.e., stdout has been written to). Is there any way to determine that?
I know I could also write to a variable and test whether it's empty, but I'd like to avoid a variable if I can.
I think that will do what you need. If you echo something between # THE SCRIPT ITSELF and # END, THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT will be printed STDOUT HAS NOT BEEN TOUCHED else...
#!/bin/bash
readonly TMP=$(mktemp /tmp/test_XXXXXX)
exec 3<> "$TMP" # open tmp file as fd 3
exec 4>&1 # save current value of stdout as fd 4
exec >&3 # redirect stdout to fd 3 (tmp file)
# THE SCRIPT ITSELF
echo Hello World
# END
exec >&4 # restore save stdout
exec 3>&- # close tmp file
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo "THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT"
echo
cat "$TMP"
else
echo "STDOUT HAS NOT BEEN TOUCHED"
fi
rm "$TMP"
So, output of the script as is:
THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT
Hello World
and if you remove the echo Hello World line:
STDOUT HAS NOT BEEN TOUCHED
And if you really want to test that while running the script itself, you can do that, too :-)
#!/bin/bash
#FIRST ELSE
function echo_fl() {
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo $2
else
echo $1
fi
}
TMP=$(mktemp /tmp/test_XXXXXX)
exec 3 "$TMP" # open tmp file as fd 3
exec 4>&1 # save current value of stdout as fd 4
exec >&3 # redirect stdout to fd 3 (tmp file)
# THE SCRIPT ITSELF
for f in fst snd trd; do
echo_fl "$(echo $f | tr a-z A-Z)" "$f"
done
# END
exec >&4 # restore save stdout
exec 3>&- # close tmp file
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo "THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT"
echo
cat "$TMP"
else
echo "STDOUT HAS NOT BEEN TOUCHED"
fi
rm "$TMP"
output is:
THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT
FST
snd
trd
as you can see: Only the first line (FST) has caps on. That's what the echo_fl function does for you: If it's the first line of output, if echoes the first argument, if it's not it echoes the second argument :-)
It's hard to tell what you are trying to do here, but if your script is printing to stdout, you could simply pipe it to perl:
yourcommand | perl -pe 'if ($. == 1) { print "First line is: $_" }'
It all depends on what kind of changes you are attempting to do.
You cannot use the -f option with %z. The line TMP_SIZE=$(stat -f %z "$TMP") produces a long string that fails the test in if [ $TMP_SIZE -gt 0 ].

Bash variable scope

Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)

Resources