I want to go through a text file, that contains one command per line. I want to read each line, execute each command, and then save the output to a file.
The part of the script that's giving me problems is:
echo COMMAND LOG > $dest/command.log
echo ====== >> $dest/command.log
while IFS= read -r v
do
echo ====== >> $dest/command.log
echo COMMAND: $v >> $dest/command.log
echo ======OUTPUT====== >> $dest/command.log
$v >> $dest/command.log
echo ====== >> $dest/command.log
done < "$commands"
It works great for everything except "echo $PATH" which generates:
======
COMMAND: echo $PATH
======OUTPUT======
$PATH
Is there anyway to get it to work properly? Thanks in advance.
It's expected, because you are fetching a file line by line, you got strings, not variables. $v is a "$PATH" string, that's all.
You must eval your string to get associated variable's value:
echo COMMAND: $(eval $v) >> $dest/command.log
Related
I am trying to read a text file which has few commented starts with '#', my bash script should read the lines of the text file which doesn't start with '#'.
Also im trying to capture the output of echo statements in both logs and to show it console window for the user understanding.
I have tried to use the below query for capturing logs and printing in console
exec 2>&1 1>>$logfile
For reading each line of the file and calling the function, i have declared an array and to eliminate lines which starts with '#' , i have used the below query.
declare -a cmd_array
while read -r -a cmd_array | grep -vE '^(\s*$|#)'
do
"${cmd_array[#]}"
done < "$text_file"
Note : I need to eliminate the line starts with '#' and remaining lines to be read and place in array as declared.
Bash script
***********
#! /bin/bash
Function_1()
{
now=$( date '+%Y%m%d%H%M' )
eval logfile="$1"_"$now".log
exec 2>&1 1>>$logfile ### Capture echo output in log and printing in console
#exec 3>&1 1>>$logfile 2>&1
echo " "
echo "############################"
echo "Function execution Begins"
echo "############################"
echo "Log file got created with file name as $1.log"
eval number=$1
eval path=$2
echo "number= $number"
ls -lR $path >> temp.txt
if [ $? -eq 0 ]; then
echo " Above query executed."
else
echo "Query execution failed"
fi
echo "############################"
echo "Function execution Ends"
echo "############################"
echo " "
}
text_file=$1
echo $text_file
declare -a cmd_array ### declaring a array
while read -r -a cmd_array | grep -vE '^(\s*$|#)' ### Read each line in the file with doesnt starts with '#' & keep it in array
do
"${cmd_array[#]}"
done < "$text_file"
Text file
*********
####################################
#Test
#Line2
####################################
Function_1 '125' '' ''
Function_1 '123' '' ''
Consider piping the grep output into the read:
declare -a cmd_array ### declaring a array
### Read each line in the file with doesnt starts with '#' & keep it in array
grep -vE '^(\s*$|#)' < "$text_file" | while read -r -a cmd_array
do
"${cmd_array[#]}"
done
I'm not clear about the output/logging comment. If you need the output appended to a file, in addition to stdout/console), consider using the 'tee' (probably 'tee -a')
I tested with the input file inputfile
echo a
Function_1 '125' '' ''
# skip me
Function_1 '123' '' ''
echo b
and wrote this script:
declare -a cmd_array ### declaring a array
while read -r -a cmd_array
do
echo "${cmd_array[#]}"
"${cmd_array[#]}"
echo
done < <(grep -vE '^(\s*$|#)' inputfile)
For showing output in log and console, see https://unix.stackexchange.com/a/145654/57293
As #GordonDavisson suggested in a comment, you get a simular result with
source inputfile
ignoring comments and empty lines, and calling functions, so I am not sure why you would want an array. This command can be included in your master script, you do not need to modify the inputfile.
Another advantage of sourcing the input is the handling of multi-line input and # in strings:
Function_1 '123' 'this is the second parameter, the third will be on the next line' \
'third parameter for the Function_1 call'
echo "This echo continues
on the next line."
echo "Don't delete # comments in a string"
Function_1 '124' 'Parameter with #, interesting!' ''
I'm trying to edit my working bash script to an SGE script in order to submit it as a job to the cluster.
Currently I have:
#!/bin/bash
# Perform fastqc on files in a specified directory.
for ((j=1; j <=17; j++))
do
directory=/data4/una/batch"$j"/
files=$""$directory"/*.fastq.gz"
batch=$"batch_"$j""
outfile=$""$batch"_submit_script.sh"
echo "#!/bin/bash">>$outfile;
echo "# Your job name">>$outfile;
echo "# -N $batch">>$outfile;
echo "# The job should be placed into the queue 'all.q'">>$outfile;
echo "#$ -q all.q">>$outfile;
echo "# Running in the current working directory">>$outfile;
echo "#$ -cwd">>$outfile;
echo "">>$outfile;
echo "# Export some necessary environment variables">>$outfile;
echo "#$ -S /bin/bash">>$outfile;
echo "#$ -v PATH">>$outfile;
echo "#$ -v LD_LIBRARY_PATH">>$outfile;
echo "#$ -v PYTHONPATH">>$outfile;
echo "# Finally, put your command here">>$outfile;
echo "">>$outfile;
echo "#$ for i in $files;">>$outfile;
echo "#$ do;">>$outfile;
echo "#$ fastqc -f fastq -o /data4/una/test/fastq/$i;">>$outfile;
echo "#$done">>$outfile;
echo "">>$outfile;
qsub $outfile;
done
But I'm getting an error:
Unable to read script file because of error: ERROR! invalid option argument "-f"
But
fastqc -f fastq -o /data4/una/test/fastq/$i
is a totally valid line in my bash script.
Thoughts?
Thanks!
It actually was poor formatting for my loop that was causing this error. I didn't need to start those lines with #$ at all, so those lines become:
echo "for i in $files;">>$outfile;
echo "do">>$outfile;
echo " fastqc -f fastq -o /data4/una/test/fastqc $i">>$outfile;
echo "done">>$outfile;
echo "">>$outfile;
qsub $outfile;
Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)
When as part of shell script only one line is operating on a file using sed command the redirected file contains the updated data, as below
cat ${PROP_PATH}/${PROP_FILE} | sed "s!${ISTR_KEY}=.*!${ISTR_KEY}=${SIM_ISTR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
whereas when it is executed as part of a shell script, where after this another sed command updates the same file as in the below script at the end what i get is an empty file, why ? ..... ideas please.
(check 'switchAll2Sim()' function below)
#!/bin/ksh
#
SIM_ICR_KEY_VAL="http://www.example.com/sim/http/icr"
SIM_ISTR_KEY_VAL="http://www.example.com/sim/http/istr"
SIM_GT_KEY_VAL="http://www.example.com/sim/http/gtr"
#
ICR_KEY="interface.url.icr"
ISTR_KEY="interface.url.istr"
GT_KEY="interface.ws.url.gt"
## Property Files
PROP_PATH=""
PROP_FILE="properties"
##
DATE=`date +%m%d%Y`
DATETIME=`date +%m%d%Y-%T`
BCKUP_DIR=_bckup
UPDATEDPROPS_DIR=_updatedprops
# ----------------------------------
pause(){
echo "Press [Enter] key to continue..."
read fackEnterKey
}
permissions(){
chmod 777 ${UPDATEDPROPS_DIR}
}
backup(){
if [ ! -d "${BCKUP_DIR}" ]; then
mkdir ${BCKUP_DIR}
fi
if [ ! -d "${UPDATEDPROPS_DIR}" ]; then
mkdir ${UPDATEDPROPS_DIR}
fi
permissions
## keep backup of properties
cp ${PROP_PATH}/${PROP_FILE} ${BCKUP_DIR}/${PROP_FILE}_${DATETIME}
echo "Backup of property files completed at: " ${DATETIME}
}
#-------------------------------------------------------------
# switch all properties to SIM
#-------------------------------------------------------------
switchAll2Sim(){
backup
#
# update files
cat ${PROP_PATH}/${PROP_FILE} | sed "s!${ISTR_KEY}=.*!${ISTR_KEY}=${SIM_ISTR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
cat ${UPDATEDPROPS_DIR}/${PROP_FILE} | sed "s!${ICR_KEY}=.*!${ICR_KEY}=${SIM_ICR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
cat ${UPDATEDPROPS_DIR}/${PROP_FILE} | sed "s!${GT_KEY}=.*!${GT_KEY}=${SIM_GT_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
echo "Switch all to SIM completed at: " ${DATETIME}
pause
}
# switch all properties to real
#-------------------------------------------------------------
switchAll2Real(){
pause
}
#-------------------------------------------------------------
dispCurrentStats(){
echo "Displaying current properties..."
echo "*********************************"
echo " File: " ${PROP_PATH}/${PROP_FILE}
grep ${ICR_KEY} ${PROP_PATH}/${PROP_FILE}
grep ${ISTR_KEY} ${PROP_PATH}/${PROP_FILE}
grep ${GT_KEY} ${PROP_PATH}/${PROP_FILE}
#
echo "*********************************"
pause
}
show_menus() {
clear
echo "~~~~~~~~~~~~~~~~~~~~~"
echo " M E N U"
echo "~~~~~~~~~~~~~~~~~~~~~"
echo "1. Display current properties"
echo "2. Switch all to real"
echo "3. Switch all to simulator"
echo "4. Exit"
}
# read input from the keyboard and take a action
read_options(){
read option
case $option in
1) dispCurrentStats ;;
2) switchAll2Real ;;
3) switchAll2Sim ;;
4) exit 0;;
*) echo "Please insert options 1 ~ 4";;
esac
}
# -----------------------------------
# Main - infinite loop
# ------------------------------------
while true
do
show_menus
read_options
done
Thanks, using '-i, says [sed: illegal option -- i]
Then you have to work with tmp files.
cp foo foo.tmp
sed "s/x/y/" foo.tmp > foo
/bin/rm foo.tmp
OR
sed "s/x/y/" foo > foo.tmp
/bin/mv -f foo.tmp foo
is probably more efficient.
I hope this helps.
Your problem is that cat is reading from the same file that sed is writing to.
cat foo | sed "s/x/y/" > foo
Will not work because cat and sed run at the same time, not one after the other.
To fix this try the -i option to sed.
sed -i "s/x/y/" foo
Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)