multiple sed operations creating empty file - shell

When as part of shell script only one line is operating on a file using sed command the redirected file contains the updated data, as below
cat ${PROP_PATH}/${PROP_FILE} | sed "s!${ISTR_KEY}=.*!${ISTR_KEY}=${SIM_ISTR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
whereas when it is executed as part of a shell script, where after this another sed command updates the same file as in the below script at the end what i get is an empty file, why ? ..... ideas please.
(check 'switchAll2Sim()' function below)
#!/bin/ksh
#
SIM_ICR_KEY_VAL="http://www.example.com/sim/http/icr"
SIM_ISTR_KEY_VAL="http://www.example.com/sim/http/istr"
SIM_GT_KEY_VAL="http://www.example.com/sim/http/gtr"
#
ICR_KEY="interface.url.icr"
ISTR_KEY="interface.url.istr"
GT_KEY="interface.ws.url.gt"
## Property Files
PROP_PATH=""
PROP_FILE="properties"
##
DATE=`date +%m%d%Y`
DATETIME=`date +%m%d%Y-%T`
BCKUP_DIR=_bckup
UPDATEDPROPS_DIR=_updatedprops
# ----------------------------------
pause(){
echo "Press [Enter] key to continue..."
read fackEnterKey
}
permissions(){
chmod 777 ${UPDATEDPROPS_DIR}
}
backup(){
if [ ! -d "${BCKUP_DIR}" ]; then
mkdir ${BCKUP_DIR}
fi
if [ ! -d "${UPDATEDPROPS_DIR}" ]; then
mkdir ${UPDATEDPROPS_DIR}
fi
permissions
## keep backup of properties
cp ${PROP_PATH}/${PROP_FILE} ${BCKUP_DIR}/${PROP_FILE}_${DATETIME}
echo "Backup of property files completed at: " ${DATETIME}
}
#-------------------------------------------------------------
# switch all properties to SIM
#-------------------------------------------------------------
switchAll2Sim(){
backup
#
# update files
cat ${PROP_PATH}/${PROP_FILE} | sed "s!${ISTR_KEY}=.*!${ISTR_KEY}=${SIM_ISTR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
cat ${UPDATEDPROPS_DIR}/${PROP_FILE} | sed "s!${ICR_KEY}=.*!${ICR_KEY}=${SIM_ICR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
cat ${UPDATEDPROPS_DIR}/${PROP_FILE} | sed "s!${GT_KEY}=.*!${GT_KEY}=${SIM_GT_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
echo "Switch all to SIM completed at: " ${DATETIME}
pause
}
# switch all properties to real
#-------------------------------------------------------------
switchAll2Real(){
pause
}
#-------------------------------------------------------------
dispCurrentStats(){
echo "Displaying current properties..."
echo "*********************************"
echo " File: " ${PROP_PATH}/${PROP_FILE}
grep ${ICR_KEY} ${PROP_PATH}/${PROP_FILE}
grep ${ISTR_KEY} ${PROP_PATH}/${PROP_FILE}
grep ${GT_KEY} ${PROP_PATH}/${PROP_FILE}
#
echo "*********************************"
pause
}
show_menus() {
clear
echo "~~~~~~~~~~~~~~~~~~~~~"
echo " M E N U"
echo "~~~~~~~~~~~~~~~~~~~~~"
echo "1. Display current properties"
echo "2. Switch all to real"
echo "3. Switch all to simulator"
echo "4. Exit"
}
# read input from the keyboard and take a action
read_options(){
read option
case $option in
1) dispCurrentStats ;;
2) switchAll2Real ;;
3) switchAll2Sim ;;
4) exit 0;;
*) echo "Please insert options 1 ~ 4";;
esac
}
# -----------------------------------
# Main - infinite loop
# ------------------------------------
while true
do
show_menus
read_options
done

Thanks, using '-i, says [sed: illegal option -- i]
Then you have to work with tmp files.
cp foo foo.tmp
sed "s/x/y/" foo.tmp > foo
/bin/rm foo.tmp
OR
sed "s/x/y/" foo > foo.tmp
/bin/mv -f foo.tmp foo
is probably more efficient.
I hope this helps.

Your problem is that cat is reading from the same file that sed is writing to.
cat foo | sed "s/x/y/" > foo
Will not work because cat and sed run at the same time, not one after the other.
To fix this try the -i option to sed.
sed -i "s/x/y/" foo

Related

Handling logs of bash script and comments in text file

I am trying to read a text file which has few commented starts with '#', my bash script should read the lines of the text file which doesn't start with '#'.
Also im trying to capture the output of echo statements in both logs and to show it console window for the user understanding.
I have tried to use the below query for capturing logs and printing in console
exec 2>&1 1>>$logfile
For reading each line of the file and calling the function, i have declared an array and to eliminate lines which starts with '#' , i have used the below query.
declare -a cmd_array
while read -r -a cmd_array | grep -vE '^(\s*$|#)'
do
"${cmd_array[#]}"
done < "$text_file"
Note : I need to eliminate the line starts with '#' and remaining lines to be read and place in array as declared.
Bash script
***********
#! /bin/bash
Function_1()
{
now=$( date '+%Y%m%d%H%M' )
eval logfile="$1"_"$now".log
exec 2>&1 1>>$logfile ### Capture echo output in log and printing in console
#exec 3>&1 1>>$logfile 2>&1
echo " "
echo "############################"
echo "Function execution Begins"
echo "############################"
echo "Log file got created with file name as $1.log"
eval number=$1
eval path=$2
echo "number= $number"
ls -lR $path >> temp.txt
if [ $? -eq 0 ]; then
echo " Above query executed."
else
echo "Query execution failed"
fi
echo "############################"
echo "Function execution Ends"
echo "############################"
echo " "
}
text_file=$1
echo $text_file
declare -a cmd_array ### declaring a array
while read -r -a cmd_array | grep -vE '^(\s*$|#)' ### Read each line in the file with doesnt starts with '#' & keep it in array
do
"${cmd_array[#]}"
done < "$text_file"
Text file
*********
####################################
#Test
#Line2
####################################
Function_1 '125' '' ''
Function_1 '123' '' ''
Consider piping the grep output into the read:
declare -a cmd_array ### declaring a array
### Read each line in the file with doesnt starts with '#' & keep it in array
grep -vE '^(\s*$|#)' < "$text_file" | while read -r -a cmd_array
do
"${cmd_array[#]}"
done
I'm not clear about the output/logging comment. If you need the output appended to a file, in addition to stdout/console), consider using the 'tee' (probably 'tee -a')
I tested with the input file inputfile
echo a
Function_1 '125' '' ''
# skip me
Function_1 '123' '' ''
echo b
and wrote this script:
declare -a cmd_array ### declaring a array
while read -r -a cmd_array
do
echo "${cmd_array[#]}"
"${cmd_array[#]}"
echo
done < <(grep -vE '^(\s*$|#)' inputfile)
For showing output in log and console, see https://unix.stackexchange.com/a/145654/57293
As #GordonDavisson suggested in a comment, you get a simular result with
source inputfile
ignoring comments and empty lines, and calling functions, so I am not sure why you would want an array. This command can be included in your master script, you do not need to modify the inputfile.
Another advantage of sourcing the input is the handling of multi-line input and # in strings:
Function_1 '123' 'this is the second parameter, the third will be on the next line' \
'third parameter for the Function_1 call'
echo "This echo continues
on the next line."
echo "Don't delete # comments in a string"
Function_1 '124' 'Parameter with #, interesting!' ''

How can I use process substitution strings in BASH?

I know I can do something like
cat <(cat somefile)
But I want to build up a string of <().
So:
for file in *.file; do
mySubs="${mySubs} <(cat ${file})"
done
cat ${mySubs} #cat <(cat 1.file) <(cat 2.file) ... <(cat something.file)
Without having to use eval.
Use named pipes directly. Use mktemp to create temporary file names for each pipe so that you can remove them after you are done.
fifos=()
for f in file1 file2 file3; do
t=$(mktemp)
mkfifo "$t"
pipes+=("$t")
someCommand "$f" > "$t" &
done
someOtherCommand "${pipes[#]}"
rm "${pipes[#]}"
I'm assuming cat is a standin for a more complicated command. Here, I'm explicitly wrapping it to show that:
#!/usr/bin/env bash
someCommand() { echo "Starting file $1"; cat "$1"; echo "Ending file $1"; }
wrap_all() {
## STAGE 1: Assemble the actual command we want to run
local fd cmd_len retval
local -a cmd fds fd_args
cmd_len=$1; shift
while (( cmd_len > 0 )); do
cmd+=( "$1" )
cmd_len=$((cmd_len - 1))
shift
done
## STAGE 2: Open an instance of someCommand for each remaining argument
local fd; local -a fds
fds=( )
for arg; do
exec {fd}< <(someCommand "$arg")
fds+=( "$fd" )
fd_args+=( "/dev/fd/$fd" )
done
## STAGE 3: Actually run the command
"${cmd[#]}" "${fd_args[#]}"; retval=$?
## STAGE 4: Close all the file descriptors
for fd in "${fds[#]}"; do
exec {fd}>&-
done
return "$retval"
}
Invocation as:
echo "one" >one.txt; echo "two" >two.txt
wrap_all 1 cat one.txt two.txt
...which outputs:
Starting file one.txt
one
Ending file one.txt
Starting file two.txt
two
Ending file two.txt
Note that this requires bash 4.1 for automatic FD allocation support (letting us avoid the need for named pipes).

How to execute ksh script from bash

I currently have a ksh script which invokes another ksh script. The "parent" ksh script needs to be invoked from a bash shell in the context of the ksh shell user. Trying the following throws back this error message
As user root in the bash shell
su - whics -c '/usr/bin/nohup /whics/t99/wv.4gm/wv99b.4gs/wv99b.sh -s 1 -m u -sleep 5 > ./nohup.out &'
/whics/t99/wv.4gm/wv99b.4gs/wv99b.sh[8]: .: wh_setENV.sh: cannot open [No such file or directory]
wh_setENV.sh is actually in /whics/t99/bin
However, when running the below commands in order I do not get this error
server:~ su - whics
server:/whics/t99 cd ./wv.4gm/wv99b.4gs
server:/whics/t99/wv.4gm/wv99b.4gs nohup ./wv99b.sh -s 1 -m u -sleep 5 &
server:/whics/t99/wv.4gm/wv99b.4gs nohup: ignoring input and appending output to `/home/whics/nohup.out'
[1] + Done nohup ./wv99b.sh -s 1 -m u -sleep 5 &
server:/whics/t99/wv.4gm/wv99b.4gs cat /home/whics/nohup.out Mon Sep 17 12:27:40 AEST 2018 : Start wv99b
wv99b.sh
#!/bin/ksh
# Copyright (C) 1992-1997 Wacher Pty. Limited
# Sccsid: %Z% %M%%Y% %Q%%I% %E%
myname=${0##*/} # a useful identifying variable
mydir=${0%$myname} # where this script is
vSFX=${myname##*.}
. wh_setENV.sh # P4813 - when using 4js:WebServices, the $fglidir/lib in LD_LIBRARY_PATH causes problems
test $debugxv && set -xv
#--------------------------------------------------------------------------------------------------------------------------------------#
wv99b_msg() {
vERR="`date` : ${vMSG}"
echo $vERR | tee -a ${vLOG}
}
#--------------------------------------------------------------------------------------------------------------------------------------#
wv99b_sysFragments() {
vSYSFRAGOK="0"
vSYSFRAGMENTS="${vTABNAME}.sysfrags.unl" ; rm -f $vSYSFRAGMENTS
$WH_ISQL $company - <<! 2>/dev/null | sed "/exprtext/d;/^$/d;s/ //g;s/[()]//g" |cut -f1 -d'=' >| ${vSYSFRAGMENTS}
select F.exprtext
from systables S, sysfragments F
where S.tabid > 99
and S.tabtype = "T"
and S.tabname = "${vTABNAME}"
and S.tabid = F.tabid
and S.tabtype = F.fragtype
and F.evalpos = 0
;
!
if [ -s ${vSYSFRAGMENTS} ] ; then
# search for the vCOLUMN in the vSYSFRAGMENTS output
vSYSFRAGOK=`grep -i ${vKEY} ${vSYSFRAGMENTS} 2>/dev/null | wc -l | awk '{print $1}'`
else
vSYSFRAGOK="0"
rm -f ${vSYSFRAGMENTS} # cleanup
fi
}
# MAIN #
vARGS="$#"
vHERE=`pwd`
vLOG="${vHERE}/errlog"
vD=0 # debug indicator
vI=0 # infile indicator
vQ=0 # email indicator
vM=0 # mode indicator
vS=0 # serial indicator
vNO_MULTI=0 # default to false
vNO_PROGI=0 # default to false
vTABLE=0 # default to 0
vSLEEP=5 # default to 0
for i in $vARGS
do
case "$i" in
-debug) vD=$2 ;;
-infile) vI=$2 ;;
-table) vTABLE=$2 ;;
-sleep) vSLEEP=$2 ;;
-no_multi) vNO_MULTI=$2 ;;
-no_progi) vNO_PROGI=$2 ;;
-m) vM=$2 ;;
-q) vQ=$2 ;;
-s) vS=$2 ;;
esac
shift
done
[[ ${vS} -eq 0 ]] && vMSG="-s parameter not supplied" && wv99b_msg && exit 1
vHERE=`pwd`
if [ ${vD} -eq 1 ] ; then
vDEBUG=" -debug 1"
else
vDEBUG=""
fi
if [ ${vI} -eq 0 ] ; then
vINFILE="wv99b.in"
else
vINFILE="${vI}"
fi
# INIT
vWVI="wv99b_I" # the name of the (I)dentify script
vWVIS="${vWVI}_${vS}" # the name of the (I)dentify script PLUS SERIAL
vWVIO="${vWVIS}.unl" # the name of the (I)dentify script
rm -f ${vWVIO}
# Check that transaction-logging is off
# check that vINFILE exists
if [ ! -s "${vINFILE}" ] ; then
vMSG="Error cannot read input file $vINFILE" ; wv99b_msg ; exit 1
fi
# Process only one(1) table
if [ ${vTABLE} != "0" ] ; then
vTABLE_FILTER=" -table ${vTABLE} "
else
vTABLE_FILTER=""
fi
# We need to check if we are running client/server
#
vDB=`echo $company | awk 'BEGIN {FS="#" } { print $1 }'`
vDBSRV=`echo $company | awk 'BEGIN {FS="#" } { print $2 }'`
case X${vDBSRV}X in
XX) vREMOTE_DB="" ;;
*) vREMOTE_DB=" -db ${vDB} -dbsrv ${vDBSRV} " ;;
esac
#_end
vMSG="Start wv99b" ; wv99b_msg
So in the wv99b.sh file, I changed
. wh_setENV.sh
to
. /whics/t99/bin/wh_setENV.sh
However, now I get the error
cannot read input file wv99b.in
I checked wv99b.in and it is in the same directory as 'wv99b.sh' (i.e. /whics/t99/wv.4gm/wv99b.4gs/ )
wh_setENV.sh
#!/usr/bin/ksh
test $debugxv && set -xv
trap door 1 2 3 5 9 15
#---------------------------------------------------------------------#
door() {
echo "`date` ERROR($?) occured in $0" >> $WH/batch.4gm/trap.log
} #end door
#---------------------------------------------------------------------#
# Script to set Environment variables for various scripts
# Stef
# Unix specific
umask 002
: ${WH:="/whics/prod"}
set -a
TERM=xterm
vHERE=`pwd`
TERMCAP=$WH/etc/termcap
vHOST=`hostname | cut -f1 -d'.'`
set +a
#LD_LIBRARY_PATH="$WH/lib.4gm/S_lib:$fglibdir/S_lib" # GUC R481
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$INFORMIXDIR/lib/c++:$INFORMIXDIR/lib/cli:$INFORMIXDIR/lib/client:$INFORMIXDIR/lib/csm:$INFORMIXDIR/lib/dmi"
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$INFORMIXDIR/lib:$INFORMIXDIR/lib/esql:$INFORMIXDIR/lib/tools.$CCODE"
export LD_LIBRARY_PATH
# EOF #
UPDATE: After OP modified/updated the question ...
wv99b.in is referenced in wv99b.sh
you don't provide the path for wv99b.in so ...
if you invoke wv99b.sh from some directory other than /whics/t99/wv.4gm/wv99b.4gs ...
when you run your test [ ! -s "${vINFILE}" ] your script (correctly) determines that "${vINFILE}" is not located in the directory where you invoked wv99b.sh from
Net result ... the original problem with sh_setENV.sh has been fixed but now the same exact problem is occurring for wv99b.in, with the same solution needed here ... invoke wv99b.sh from its home directory or provide the full path to wv99b.in.
ORIGINAL POST:
Expanding on Andre's comment ...
In wv99b.sh you have the following line:
. wh_setENV.sh
This is going to look for wh_setENV.sh in the directory where wv99b.sh is invoked from.
In your original example you've provided the full path to wv99b.sh but the question is ... what directory is that call invoked from? We can tell from the error ...
wh_setENV.sh: cannot open [No such file or directory]
... that wv99b.sh was not invoked from /whics/t99/wv.4gm/wv99b.4gs otherwise it would have found wh_setENV.sh.
From your second example it appears that the full path to wh_setENV.sh is: /whics/t99/wv.4gm/wv99b.4gs/wh_setENV.sh so you have a couple options:
in your initial example make sure you cd /whics/t99/wv.4gm/wv99b.4gs before executing wv99b.4gs
or
update wv99b.4gs to include the full path to the location of sh_setENV.sh, eg:
. /whics/t99/wv.4gm/wv99b.4gs/wh_setENV.sh

define "$key" use it in a script, but also store it

How can I create a specific line in another file using bash please? Like
echo "Please input the days you want to keep "
$key= ?
touch .beebrc; keep="$key"
where the file ".beebrc" has a line 'keep= x' and "$key" is created in the main script.
But how do I define "$key" please? And write it into ".beebrc" as a new line at position/line 8? The full function is -
function trim {
echo;
read -t "$temi" -n1 -p ""$bldgrn" Do you want to delete some of your download history? [y/n/q/r] $(tput sgr0)" ynqr ;
case "$ynqr" in
[Yy]) echo
read -t "$temi" -n3 -p ""$bldgrn" Please input the days you want to keep $(tput sgr0)" key ## ask
if test -e .beebrc && grep -q "^keep=" .beebrc 2>/dev/null ; then
sed -i "s/^keep=.*/keep=$key/" .beebrc
else
echo "keep=$key" >> .beebrc
#fi
cd /home/$USER/.get_iplayer
eval "$player" --trim-history "$key"; cd; ques;
#echo;;
[Nn]) ques;;
[Qq]) endex;;
[Rr]) exec "$beeb";;
* ) echo ""$bldgrn" Thank you $(tput sgr0)";;
esac
fi
};
Does this help in defining it all? (Sorry, should've put it in at first)
Perhaps:
read -p "Please input the days you want to keep: " key ## Ask.
echo "keep=\"$key\"" > .beebrc ## Store.
Use read to capture user input into a variable, and then write it to your file.
For example:
echo "Please input the days you want to keep "
read key
echo $key > .beebrc
#!/bin/bash
read -p "Please input the days you want to keep: " key
if test -e .beebrc && grep -q "^keep=" .beebrc 2>/dev/null ; then
sed -i "s/^keep=.*/keep=$key/" .beebrc
else
echo "keep=$key" >> .beebrc
fi
This script:
Prompts for input and stores the value in $key
Tests if .beebrc exists and that a line beginning "keep=" exists in it. If so, replace the keep= line with keep=$key
Otherwise append a new line/create the file with keep=$key.
This will need validation added because user input should not be trusted. (this answer might help)

Why does inotitywait with "echo '---' >> file" only emit an "OPEN" event?

Why does echo "---" >> file only generate a ./ OPEN file event? Shouldn't it generate other events too?
For example:
# In a script:
while true; do
change=$(inotifywait -q -r . )
echo $change
done
# Somewhere else:
echo "--" >> file
echo "--" >> file
echo "--" >> file
it is exiting after the first event and starting the loop over, by the time the first watch is removed and the second added, the events have already transpired. You can try:
inotifywait -q -m -r . | while read CHANGE; do
case "$CHANGE" in
*)echo $CHANGE;;
esac
done
the -m will continue monitoring after the first event

Resources