I have the while loop in ksh which reads the file and loop through each line.
Here is the same file contents (TestCases.txt)
TEST_PROC_1(1)/TEST_1,TEST_2,TEST_3/N/P
TEST_PROC_1(1)/TEST_1,TEST_2,TEST_3/N/N
TEST_PROC_2('CICD_DEMO.txt')/TEST_1,TEST_2,TEST_3/N/N
TEST_FUNC_1(100)/TEST_1,TEST_2,TEST_3/N/P
TEST_FUNC_2/TEST_1,TEST_2,TEST_3/N/N
TEST_PROC_4/TEST_1,TEST_2/N/N
TEST_FUNC_3(3)//N/P
The scripts which reads the document
swd=$(pwd)
export swd
file=${swd}/TestCases.txt
export testCaseIndex=1
export validateTblIndex=1
cat ${file} | while IFS=\/ read procname tablelist hold_data testcase_type
do
echo "$procname $tablelist $hold_data $testcase_type"
ksh ${swd}/assets/sh/main.sh "${procname}" "${tablelist}" "${hold_data}" "${testcase_type}" "${testCaseIndex}" "${validateTblIndex}"
ret=$?
echo $ret
(( testCaseIndex+=1 ))
(( validateTblIndex+=1 ))
done
Here is the problem
If I comment the ksh call it iterates till last line.
TEST_PROC_1(1) TEST_1,TEST_2,TEST_3 N P
0
TEST_PROC_1(1) TEST_1,TEST_2,TEST_3 N N
0
TEST_PROC_2('CICD_DEMO.txt') TEST_1,TEST_2,TEST_3 N N
0
TEST_FUNC_1(100) TEST_1,TEST_2,TEST_3 N P
0
TEST_FUNC_2 TEST_1,TEST_2,TEST_3 N N
0
TEST_PROC_4 TEST_1,TEST_2 N N
0
TEST_FUNC_3(3) N P
0
If I uncomment it stops with first line of the file.
TEST_PROC_1(1) TEST_1,TEST_2,TEST_3 N P
0
Kindly help out what could be possible issues. ksh call works fine even I run separately.
I have ksh93 version.
main.sh is also reading from standard input, which it inherits from the loop, so it is consuming data meant for the read command. Given that this surprises you, you may simply be able to redirect the script's standard input from /dev/null.
(Also, unless cat ${file} is just filling in for some other process that produces the data, use input redirection instead of a pipe.)
while IFS=/ read procname tablelist hold_data testcase_type
do
echo "$procname $tablelist $hold_data $testcase_type"
ksh ${swd}/assets/sh/main.sh "${procname}" \
"${tablelist}" "${hold_data}" "${testcase_type}" \
"${testCaseIndex}" "${validateTblIndex}" < /dev/null
ret=$?
echo $ret
(( testCaseIndex+=1 ))
(( validateTblIndex+=1 ))
done < $file
If main.sh does need to read from standard input, use a different file descriptor for read command.
while IFS=/ read procname tablelist hold_data testcase_type <&3
do
echo "$procname $tablelist $hold_data $testcase_type"
ksh ${swd}/assets/sh/main.sh "${procname}" \
"${tablelist}" "${hold_data}" "${testcase_type}" \
"${testCaseIndex}" "${validateTblIndex}" < /dev/null
ret=$?
echo $ret
(( testCaseIndex+=1 ))
(( validateTblIndex+=1 ))
done 3< $file
Related
I have a script that will execute a PHP file multiple times:
#!/bin/bash
FILE=$(cat $1)
while IFS= read -r i in $file; do
php x.php "$i" &
done < "$1"
the text will be
a
b
c
d
What should i do to show me the line number that it's using out of how many, for example
3(current line)/200(number of total lines)
I did some research but i couldn't find anything.
You need to get the number of lines in the file first using wc, then add a counter to the loop. You're redirecting the file to the while loop, so you only need to assign the i variable while you read it:
#!/bin/bash
len=$(wc -l < "$1")
j=1
while read -r i; do
echo "$j / $len"
php x.php "$i" &
j=$(( j+1 ))
done < "$1"
I've written a bash script to truncate through a server list and perform various commands. I would like to incorporate a progress indicator like a percent complete while the script is running. I found a script online to perform this but it's not working properly and I am unsure how to utilize it.
The dialog command is not an option here as I am working with nsh
#!/bin/bash
i=0
while [[ $i -lt 11 ]]; do
##
## \r = carriage return
## \c = suppress linefeed
##
echo -en "\r$i%\c\b"
(( i=i+1 ))
sleep 1
done
echo
exit 0
For testing purposes I am only connecting to each server and echoing the hostname.
for i in $(cat serverlist.txt)
do
nexec -i hostname
done
How can I utilize the first code snipped to show the progress while going through the list of servers in the code above?
To keep track of your progress, you'll want to store the list of servers in an array, so you know how many there are:
mapfile -t servers <serverlist.txt # servers is an array
n=${#servers[#]} # how many
i=0
for server in "${servers[#]}"; do
((i++))
progress=$(( i * 100 / n ))
nexec ...
echo "you are ${progress}% complete"
done
write_status() {
done=0
total=$1
columns=${COLUMNS:-100}
# print initial, empty progress bar
for (( i=0; i<columns; i++ )); do printf '-'; done; printf '\r'
# every time we read a line, print a progress bar with one more item complete
while read -r; do
done=$(( done + 1 ))
pct=$(( done * columns / total ))
for ((i=0; i<pct; i++)); do
printf '+'
done
for ((i=pct; i<columns; i++)); do
printf '-'
done
printf '\r'
if (( done == total )); then break; fi
done
}
# read server names into an array; the below is bash 4.x syntax
readarray -t servers <serverlist.txt
# direct FD 4 to the a subshell running the write_status function
exec 4> >(write_status "${#servers[#]}")
for hostname in "${servers[#]}"; do
nexec -i "$hostname" # actually run the command
echo >&4 # ...and notify write_status that it completed
done
If you wanted to parallelize the remote commands, you can do that by replacing the for loop at the bottom with the following:
for hostname in "${servers[#]}"; do
(nexec -i "$hostname"; echo >&4) &
done
wait
If your shell is a version of bash prior to 4.0, then the readarray -t servers <serverlist can be replaced with the following:
servers=( )
while IFS= read -r server; do servers+=( "$server" ); done <serverlist.txt
for m in $count
do
`cat $op ${arr[$m]} > $op1`
`rm -f $op`
`touch $op`
`cat $op1 ${arr[$m+1]} > $op`
if [ $m ge $count ]; then
`rm -f $op1`
`touch $op1`
fi
m=$((m+1))
done
I wanted to continuously loop from the start count 2 till the end count 10 . The $count=10 here. But the above piece of code executes the for loop only once.
Rainy sunday - having much free time - long answer ;)
Many issues with your script, some recommended solutions. Because you used the construction m=$((m+1)) - will be using bash as "shell". (Consider adding the bash tag)
For the cycle - several possibilities
count=10
m=2 #start with 2
while (( $m <= $count )) #while m is less or equal to 10
do #do
echo $m #this action
let m++ #increment m (add one to m)
done #end of while
or, if the count is a constant (not a variable), you can write
for m in {2..10} #REMEMBER, will not works with a variables, like {2..$count}
do
echo "$m"
done
another variant - using the seq (man seq) command for counting
for m in $(seq 2 ${count:=10}) # ${count:=10} - defaults the $count to 10 if it is undefined
do
echo $m
done
or C-like for loop
let count=10
for ((m=2; m<=count; m++))
do
echo $m
done
All 4 loops produces:
2
3
4
5
6
7
8
9
10
so, having a right cycle now. Now add your specific actions.
The:
rm -f $op
touch $op
can be replaced by one command
echo -n > $op #echo nothing and write the "nothing" into the file
it is faster, because the echo is an bash builtin (doesn't start two external commands)
So your actions could looks like
cat $op ${arr[$m]} > $op1
echo -n > $op
cat $op1 ${arr[$m+1]} > $op
in this case, the echo is useless, because the second cat will write its output
to the $op anyway (and before write shortens the file to zero size), so this result is
identical with the above
cat $op ${arr[$m]} > $op1
cat $op1 ${arr[$m+1]} > $op
Those two cat commands can be shorted to one, using bash's >> append to file redirection
cat ${arr[$m]} ${arr[m+1]} >> $op
The whole script could look like the next
#making a testing environment
for f in $(seq 12) #create 12 files opdata-N
do
arr[$f]="opdata-$f" #store the filenames in the array "arr"
echo "data-$f" > ${arr[$f]} #each file contains one line "data-N"
done
#echo ${arr[#]}
#setting the $op and $op1 filenames
#consider choosing more descriptive variable names
op="file_op"
#op1="file_op1" #not needed
#add some initial (old) value to $op
echo "initial value" > $op
#end of creating the testing environment
#the script
count=10
for m in $(seq 2 $count)
do
cat ${arr[$m]} ${arr[m+1]} >> $op
done
at the end, file $op will contain:
initial value
data-2
data-3
data-3
data-4
data-4
data-5
data-5
data-6
data-6
data-7
data-7
data-8
data-8
data-9
data-9
data-10
data-10
data-11
BTW, are you sure about the result? Because if only want add file-2 .. file-10 to the end of $op (without duplicating entries), you can simple write:
cat file-{2..10} >> $op #the '>>' adds to the end of file...
or by using your array:
startpos=2
count=10
cat ${arr[#]:$startpos:$count} >> $op
Ufff.. ;)
Ps: usually it is good practice to enclose variables in double quotes like "$filename" - in the above examples for better readability I omitted them.
Any loop needs a "condition to keep looping". When you use a
for m in count
type of loop, the condition is "if there are more elements in the collection count, pick the next one and keep going". This doesn't seem to be what you want. You are looking for the bash equivalent of
for(m = 0; m < 10; m++)
I think. The best way to do this is - with exactly that kind of loop (but note - an extra pair of parentheses, and a semicolon):
#!/bin/bash
# Display message 5 times
for ((i = 0 ; i < 5 ; i++)); do
echo "Welcome $i times."
done
see nix craft for original
I think you can extend this to your situation… if I understood your question correctly you need something like this:
for ((m = 2; m <= 10; m++))
do
cat $op ${arr[$m]} > $op1
rm -f $op
touch $op
cat $op1 ${arr[$m+1]} > $op
if [ $m ge $count ]; then
rm -f $op1
touch $op1
fi
done
Use a while loop instead.
The for loop is when you have multiple objects to iterate against. You have only one, i.e. $count.
How would you achieve this in bash. It's a question I got asked in an interview and I could think of answers in high level languages but not in shell.
As I understand it, the real implementation of tail seeks to the end of the file and then reads backwards.
The main idea is to keep a fixed-size buffer and to remember the last lines. Here's a quick way to do a tail using the shell:
#!/bin/bash
SIZE=5
idx=0
while read line
do
arr[$idx]=$line
idx=$(( ( idx + 1 ) % SIZE ))
done < text
for ((i=0; i<SIZE; i++))
do
echo ${arr[$idx]}
idx=$(( ( idx + 1 ) % SIZE ))
done
If all not-tail commands are allowed, why not be whimsical?
#!/bin/sh
[ -r "$1" ] && exec < "$1"
tac | head | tac
Use wc -l to count the number of lines in the file. Subtract the number of lines you want from this, and add 1, to get the starting line number. Then use this with sed or awk to start printing the file from that line number, e.g.
sed -n "$start,\$p"
There's this:
#!/bin/bash
readarray file
lines=$(( ${#file[#]} - 1 ))
for (( line=$(($lines-$1)), i=${1:-$lines}; (( line < $lines && i > 0 )); line++, i-- )); do
echo -ne "${file[$line]}"
done
Based on this answer: https://stackoverflow.com/a/8020488/851273
You pass in the number of lines at the end of the file you want to see then send the file via stdin, puts the entire file into an array, and only prints the last # lines of the array.
The only way I can think of in “pure” shell is to do a while read linewise on the whole file into an array variable with indexing modulo n, where n is the number of tail lines (default 10) — i.e. a circular buffer, then iterate over the circular buffer from where you left off when the while read ends. It's not efficient or elegant, in any sense, but it'll work and avoids reading the whole file into memory. For example:
#!/bin/bash
incmod() {
let i=$1+1
n=$2
if [ $i -ge $2 ]; then
echo 0
else
echo $i
fi
}
n=10
i=0
buffer=
while read line; do
buffer[$i]=$line
i=$(incmod $i $n)
done < $1
j=$i
echo ${buffer[$i]}
i=$(incmod $i $n)
while [ $i -ne $j ]; do
echo ${buffer[$i]}
i=$(incmod $i $n)
done
This script somehow imitates tail:
#!/bin/bash
shopt -s extglob
LENGTH=10
while [[ $# -gt 0 ]]; do
case "$1" in
--)
FILES+=("${#:2}")
break
;;
-+([0-9]))
LENGTH=${1#-}
;;
-n)
if [[ $2 != +([0-9]) ]]; then
echo "Invalid argument to '-n': $1"
exit 1
fi
LENGTH=$2
shift
;;
-*)
echo "Unknown option: $1"
exit 1
;;
*)
FILES+=("$1")
;;
esac
shift
done
PRINTHEADER=false
case "${#FILES[#]}" in
0)
FILES=("/dev/stdin")
;;
1)
;;
*)
PRINTHEADER=true
;;
esac
IFS=
for I in "${!FILES[#]}"; do
F=${FILES[I]}
if [[ $PRINTHEADER == true ]]; then
[[ I -gt 0 ]] && echo
echo "==> $F <=="
fi
if [[ LENGTH -gt 0 ]]; then
LINES=()
COUNT=0
while read -r LINE; do
LINES[COUNT++ % LENGTH]=$LINE
done < "$F"
for (( I = COUNT >= LENGTH ? LENGTH : COUNT; I; --I )); do
echo "${LINES[--COUNT % LENGTH]}"
done
fi
done
Example run:
> bash script.sh -n 12 <(yes | sed 20q) <(yes | sed 5q)
==> /dev/fd/63 <==
y
y
y
y
y
y
y
y
y
y
y
y
==> /dev/fd/62 <==
y
y
y
y
y
> bash script.sh -4 <(yes | sed 200q)
y
y
y
y
Here's the answer I would give if I were actually asked this question in an interview:
What environment is this where I have bash but not tail? Early boot scripts, maybe? Can we get busybox in there so we can use the full complement of shell utilities? Or maybe we should see if we can squeeze a stripped-down Perl interpreter in, even without most of the modules that would make life a whole lot easier. You know dash is much smaller than bash and perfectly good for scripting use, right? That might also help. If none of that is an option, we should check how much space a statically linked C mini-tail would need, I bet I can fit it in the same number of disk blocks as the shell script you want.
If that doesn't convince the interviewer that it's a silly question, then I go on to observe that I don't believe in using bash extensions, because the only good reason to write anything complicated in shell script nowadays is if total portability is an overriding concern. By avoiding anything that isn't portable even in one-offs, I don't develop bad habits, and I don't get tempted to do something in shell when it would be better done in a real programming language.
Now the thing is, in truly portable shell, arrays may not be available. (I don't actually know whether the POSIX shell spec has arrays, but there certainly are legacy-Unix shells that don't have them.) So, if you have to emulate tail using only shell builtins and it's got to work everywhere, this is the best you can do, and yes, it's hideous, because you're writing in the wrong language:
#! /bin/sh
a=""
b=""
c=""
d=""
e=""
f=""
while read x; do
a="$b"
b="$c"
c="$d"
d="$e"
e="$f"
f="$x"
done
printf '%s\n' "$a"
printf '%s\n' "$b"
printf '%s\n' "$c"
printf '%s\n' "$d"
printf '%s\n' "$e"
printf '%s\n' "$f"
Adjust the number of variables to match the number of lines you want to print.
The battle-scarred will note that printf is not 100% available either. Unfortunately, if all you have is echo, you are up a creek: some versions of echo cannot print the literal string "-n", and others cannot print the literal string "\n", and even figuring out which one you have is a bit of a pain, particularly as, if you don't have printf (which is in POSIX), you probably don't have user-defined functions either.
(N.B. The code in this answer, sans rationale, was originally posted by user 'Nirk' but then deleted under downvote pressure from people whom I shall charitably assume were not aware that some shells do not have arrays.)
According the bash(1) man pages, when I run the following:
set -e
x=2
echo Start $x
while [ $((x--)) -gt 0 ]; do echo Loop $x; done | cat
echo End $x
The output will be:
Start 2
Loop 1
Loop 0
End 2
After the loop (runs as a subshell) the variable x reset to 2. But if I remove the pipe the x will be updated:
Start 2
Loop 1
Loop 0
End -1
I need to change the x but, I need the pipe too.
Any idea how to get around this problem?
bash always (at least as of 4.2) runs all non-rightmost parts of a pipeline in a subshell. If the value of x needs to change in the calling shell, you must rewrite your code to avoid the pipeline.
One horrible-looking example:
# If you commit to one bash feature, may as well commit to them all:
# Arithmetic compound: (( x-- > 0 ))
# Process substitution: > >( cat )
while (( x-- > 0 )); do echo Loop $x; done > >( cat )