I have stored some arguments value in sample.txt
1 >> sample.txt
2 >> sample.txt
3 >> sample.txt
I have tried to parse the sample.txt in a shell script file to collect and assign the values to specific variables.
#!/bin/sh
if [ -f sample.txt ]; then
cat sample.txt | while read Param
do
let count++
if [ "${count}" == 1 ]; then
Var1=`echo ${Param}`
elif [ "${count}" == 2 ]; then
Var2=`echo ${Param}`
else
Var3=`echo ${Param}`
fi
done
fi
echo "$Var1"
echo "$Var2"
echo results prints nothing. I would expect 1 and 2 should be printed. Anyone help?
You are running the while loop in a subshell; use input redirection instead of cat:
while read Param; do
...
done < sample.txt
(Also, Var1=$Param is much simpler than Var1=$(echo $Param).)
However, there's no point in use a while loop if you know ahead of time how many variables you are setting; just use the right number of read commands directly.
{ read Var1; read Var2; read Var3; } < sample.txt
Related
In the following program, if I set the variable $foo to the value 1 inside the first if statement, it works in the sense that its value is remembered after the if statement. However, when I set the same variable to the value 2 inside an if which is inside a while statement, it's forgotten after the while loop. It's behaving like I'm using some sort of copy of the variable $foo inside the while loop and I am modifying only that particular copy. Here's a complete test program:
#!/bin/bash
set -e
set -u
foo=0
bar="hello"
if [[ "$bar" == "hello" ]]
then
foo=1
echo "Setting \$foo to 1: $foo"
fi
echo "Variable \$foo after if statement: $foo"
lines="first line\nsecond line\nthird line"
echo -e $lines | while read line
do
if [[ "$line" == "second line" ]]
then
foo=2
echo "Variable \$foo updated to $foo inside if inside while loop"
fi
echo "Value of \$foo in while loop body: $foo"
done
echo "Variable \$foo after while loop: $foo"
# Output:
# $ ./testbash.sh
# Setting $foo to 1: 1
# Variable $foo after if statement: 1
# Value of $foo in while loop body: 1
# Variable $foo updated to 2 inside if inside while loop
# Value of $foo in while loop body: 2
# Value of $foo in while loop body: 2
# Variable $foo after while loop: 1
# bash --version
# GNU bash, version 4.1.10(4)-release (i686-pc-cygwin)
echo -e $lines | while read line
...
done
The while loop is executed in a subshell. So any changes you do to the variable will not be available once the subshell exits.
Instead you can use a here string to re-write the while loop to be in the main shell process; only echo -e $lines will run in a subshell:
while read line
do
if [[ "$line" == "second line" ]]
then
foo=2
echo "Variable \$foo updated to $foo inside if inside while loop"
fi
echo "Value of \$foo in while loop body: $foo"
done <<< "$(echo -e "$lines")"
You can get rid of the rather ugly echo in the here-string above by expanding the backslash sequences immediately when assigning lines. The $'...' form of quoting can be used there:
lines=$'first line\nsecond line\nthird line'
while read line; do
...
done <<< "$lines"
UPDATED#2
Explanation is in Blue Moons's answer.
Alternative solutions:
Eliminate echo
while read line; do
...
done <<EOT
first line
second line
third line
EOT
Add the echo inside the here-is-the-document
while read line; do
...
done <<EOT
$(echo -e $lines)
EOT
Run echo in background:
coproc echo -e $lines
while read -u ${COPROC[0]} line; do
...
done
Redirect to a file handle explicitly (Mind the space in < <!):
exec 3< <(echo -e $lines)
while read -u 3 line; do
...
done
Or just redirect to the stdin:
while read line; do
...
done < <(echo -e $lines)
And one for chepner (eliminating echo):
arr=("first line" "second line" "third line");
for((i=0;i<${#arr[*]};++i)) { line=${arr[i]};
...
}
Variable $lines can be converted to an array without starting a new sub-shell. The characters \ and n has to be converted to some character (e.g. a real new line character) and use the IFS (Internal Field Separator) variable to split the string into array elements. This can be done like:
lines="first line\nsecond line\nthird line"
echo "$lines"
OIFS="$IFS"
IFS=$'\n' arr=(${lines//\\n/$'\n'}) # Conversion
IFS="$OIFS"
echo "${arr[#]}", Length: ${#arr[*]}
set|grep ^arr
Result is
first line\nsecond line\nthird line
first line second line third line, Length: 3
arr=([0]="first line" [1]="second line" [2]="third line")
You are asking this bash FAQ. The answer also describes the general case of variables set in subshells created by pipes:
E4) If I pipe the output of a command into read variable, why
doesn't the output show up in $variable when the read command finishes?
This has to do with the parent-child relationship between Unix
processes. It affects all commands run in pipelines, not just
simple calls to read. For example, piping a command's output
into a while loop that repeatedly calls read will result in
the same behavior.
Each element of a pipeline, even a builtin or shell function,
runs in a separate process, a child of the shell running the
pipeline. A subprocess cannot affect its parent's environment.
When the read command sets the variable to the input, that
variable is set only in the subshell, not the parent shell. When
the subshell exits, the value of the variable is lost.
Many pipelines that end with read variable can be converted
into command substitutions, which will capture the output of
a specified command. The output can then be assigned to a
variable:
grep ^gnu /usr/lib/news/active | wc -l | read ngroup
can be converted into
ngroup=$(grep ^gnu /usr/lib/news/active | wc -l)
This does not, unfortunately, work to split the text among
multiple variables, as read does when given multiple variable
arguments. If you need to do this, you can either use the
command substitution above to read the output into a variable
and chop up the variable using the bash pattern removal
expansion operators or use some variant of the following
approach.
Say /usr/local/bin/ipaddr is the following shell script:
#! /bin/sh
host `hostname` | awk '/address/ {print $NF}'
Instead of using
/usr/local/bin/ipaddr | read A B C D
to break the local machine's IP address into separate octets, use
OIFS="$IFS"
IFS=.
set -- $(/usr/local/bin/ipaddr)
IFS="$OIFS"
A="$1" B="$2" C="$3" D="$4"
Beware, however, that this will change the shell's positional
parameters. If you need them, you should save them before doing
this.
This is the general approach -- in most cases you will not need to
set $IFS to a different value.
Some other user-supplied alternatives include:
read A B C D << HERE
$(IFS=.; echo $(/usr/local/bin/ipaddr))
HERE
and, where process substitution is available,
read A B C D < <(IFS=.; echo $(/usr/local/bin/ipaddr))
Hmmm... I would almost swear that this worked for the original Bourne shell, but don't have access to a running copy just now to check.
There is, however, a very trivial workaround to the problem.
Change the first line of the script from:
#!/bin/bash
to
#!/bin/ksh
Et voila! A read at the end of a pipeline works just fine, assuming you have the Korn shell installed.
This is an interesting question and touches on a very basic concept in Bourne shell and subshell. Here I provide a solution that is different from the previous solutions by doing some kind of filtering. I will give an example that may be useful in real life. This is a fragment for checking that downloaded files conform to a known checksum. The checksum file look like the following (Showing just 3 lines):
49174 36326 dna_align_feature.txt.gz
54757 1 dna.txt.gz
55409 9971 exon_transcript.txt.gz
The shell script:
#!/bin/sh
.....
failcnt=0 # this variable is only valid in the parent shell
#variable xx captures all the outputs from the while loop
xx=$(cat ${checkfile} | while read -r line; do
num1=$(echo $line | awk '{print $1}')
num2=$(echo $line | awk '{print $2}')
fname=$(echo $line | awk '{print $3}')
if [ -f "$fname" ]; then
res=$(sum $fname)
filegood=$(sum $fname | awk -v na=$num1 -v nb=$num2 -v fn=$fname '{ if (na == $1 && nb == $2) { print "TRUE"; } else { print "FALSE"; }}')
if [ "$filegood" = "FALSE" ]; then
failcnt=$(expr $failcnt + 1) # only in subshell
echo "$fname BAD $failcnt"
fi
fi
done | tail -1) # I am only interested in the final result
# you can capture a whole bunch of texts and do further filtering
failcnt=${xx#* BAD } # I am only interested in the number
# this variable is in the parent shell
echo failcnt $failcnt
if [ $failcnt -gt 0 ]; then
echo $failcnt files failed
else
echo download successful
fi
The parent and subshell communicate through the echo command. You can pick some easy to parse text for the parent shell. This method does not break your normal way of thinking, just that you have to do some post processing. You can use grep, sed, awk, and more for doing so.
I use stderr to store within a loop, and read from it outside.
Here var i is initially set and read inside the loop as 1.
# reading lines of content from 2 files concatenated
# inside loop: write value of var i to stderr (before iteration)
# outside: read var i from stderr, has last iterative value
f=/tmp/file1
g=/tmp/file2
i=1
cat $f $g | \
while read -r s;
do
echo $s > /dev/null; # some work
echo $i > 2
let i++
done;
read -r i < 2
echo $i
Or use the heredoc method to reduce the amount of code in a subshell.
Note the iterative i value can be read outside the while loop.
i=1
while read -r s;
do
echo $s > /dev/null
let i++
done <<EOT
$(cat $f $g)
EOT
let i--
echo $i
How about a very simple method
+call your while loop in a function
- set your value inside (nonsense, but shows the example)
- return your value inside
+capture your value outside
+set outside
+display outside
#!/bin/bash
# set -e
# set -u
# No idea why you need this, not using here
foo=0
bar="hello"
if [[ "$bar" == "hello" ]]
then
foo=1
echo "Setting \$foo to $foo"
fi
echo "Variable \$foo after if statement: $foo"
lines="first line\nsecond line\nthird line"
function my_while_loop
{
echo -e $lines | while read line
do
if [[ "$line" == "second line" ]]
then
foo=2; return 2;
echo "Variable \$foo updated to $foo inside if inside while loop"
fi
echo -e $lines | while read line
do
if [[ "$line" == "second line" ]]
then
foo=2;
echo "Variable \$foo updated to $foo inside if inside while loop"
return 2;
fi
# Code below won't be executed since we returned from function in 'if' statement
# We aready reported the $foo var beint set to 2 anyway
echo "Value of \$foo in while loop body: $foo"
done
}
my_while_loop; foo="$?"
echo "Variable \$foo after while loop: $foo"
Output:
Setting $foo 1
Variable $foo after if statement: 1
Value of $foo in while loop body: 1
Variable $foo after while loop: 2
bash --version
GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.
Though this is an old question and asked several times, here's what I'm doing after hours fidgeting with here strings, and the only option that worked for me is to store the value in a file during while loop sub-shells and then retrieve it. Simple.
Use echo statement to store and cat statement to retrieve. And the bash user must chown the directory or have read-write chmod access.
#write to file
echo "1" > foo.txt
while condition; do
if (condition); then
#write again to file
echo "2" > foo.txt
fi
done
#read from file
echo "Value of \$foo in while loop body: $(cat foo.txt)"
I am trying to write this code so that if the process reads map finished in the pipe it increments a variable by 1 so that it eventually breaks out of the while loop. Otherwise it will add unique parameters to a keys file. However it goes into an infinite loop and never breaks out of the loop.
while [ $a -le 5 ]; do
read input < map_pipe;
if [ $input = "map finished" ]; then
((a++))
echo $a
else
sort -u map_pipe >> keys.txt;
fi
done
I decided to fix it for you, not sure if this is what you wanted, but I think I am close:
#!/bin/bash
a=0 #Initialize your variable to something
while [ $a -le 5 ]; do
read input < map_pipe;
if [ "$input" = "map finished" ]; then #Put double quotes around variables to allow values with spaces
a=$(($a + 1)) #Your syntax was off, use spaces and do something with the output
else
echo $input >> keys.txt #Don't re-read the pipe, it's empty by now and sort will wait for the next input
sort -u keys.txt > tmpfile #Instead sort your file, don't save directly into the same file it will break
mv tmpfile keys.txt
#sort -u keys.txt | sponge keys.txt #Will also work instead of the other sort and mv, but sponge is not installed on most machines
fi
done
I have a series of commands chained together with pipes:
should_create_one_line | expects_one_line
The first command should_create_one_line should produce an output that only has one line, but under strange circumstances it is possible for the output to be multiline or empty.
I would like to add a step in between these two, validate_one_line:
should_create_one_line | validate_one_line | expects_one_line
If its input contains exactly 1 line then validate_one_line will simply output its input. If its input contains more than 1 line or is empty then validate_one_line should cause the whole sequence of steps to stop and return an error code.
What command can I use for validate_one_line?
Use read. Here's a shell function that meets your specs:
exactly_one_line() {
local line # Use to echo the line
read -r line || return # Guarantee at least one line is read
read && return 1 # Indicate failure if another line is successfully read
echo "$line"
}
Notes
"One line" assumes a single line followed by a newline. If your input could be like, a file with contents but no newlines, then this will fail.
Given a pipeline like a|b, a cannot prevent b from running. At a minimum, b needs to handle when a produces no output.
Demo:
$ wc -l empty oneline twolines
0 empty
1 oneline
2 twolines
3 total
$ exactly_one_line < empty; echo $?
1
$ exactly_one_line < oneline; echo $?
oneline
0
$ exactly_one_line < twolines; echo $?
1
First off, you should seriously consider adding the validation code to expects_one_line. According to this post, each process starts in its own subshell, meaning that even if validate_one_line fails, you will get an error in expects_one_line because it will try to run with no input (or a blank line). That being said, here is a bash one-liner that you can insert into your pipe to validate:
should_create_one_line.sh | ( var="$(cat)"; [ $(echo "$var" | wc -l) -ne 1 ] && exit 1 || echo "$var") | expects_one_line.sh
The problem here is that when the validation subshell returns in the exit 1 case, expects_one_line.sh will still get a single blank line. If this works for you, then great. If not, it would be better to just put the following into the beginning of expects_one_line.sh:
input="$(cat)"
[ $(echo "$var" | wc -l) -ne 1 ] && exit 1
This would guarantee that expects_one_line.sh fails properly when getting a single line without having to wonder about what the empty line that the validation outputs will do to the script.
You may find this post helpful: How to read mutliline input from stdin into variable and how to print one out in shell(sh,bash)?
You can use a bash script to check the incoming data and call the other command when the input is only 1 line
The following code starts cat when it is ONLY fet in 1 line
sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' cat
How this works
Try reading a line, if failed go to step 5
If variable $LINE is NOT empty, goto step 6
Save line inside variable $LINE
Goto step 1
If $LINE is NOT empty, goto step 7
Exit the program with status code 1
Call our program and pass our $line to it using printf
Example usage:
Printing out only if grep found 1 match:
grep .... | sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' cat
Example of the question poster:
should_create_one_line | sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' expects_one_line
I'm writing a script that searches a file, gets info that it then stores into variables, and executes a program that I made using those variables as data. I actually have all of that working, but I need to take it a step further:
What I currently have is
#!/bin/sh
START=0
END=9
LOOP=10
PASS=0
for i in $(seq 0 $LOOP)
do
LEN=$(awk '/Len =/ { print $3; exit;}' ../../Tests/shabittestvectors/SHA1ShortMsg.rsp)
MSG=$(awk '/Msg =/ { print $3; exit; }' ../../Tests/shabittestvectors/SHA1ShortMsg.rsp)
MD=$(awk '/MD =/ { print $3; exit; }' ../../Tests/shabittestvectors/SHA1ShortMsg.rsp)
echo $LEN
echo $MSG
MD=${MD:0:-1}
CIPHER=$(./cyassl hash -sha -i $MSG -l $LEN)
echo $MD
echo $CIPHER
if [ $MD == $CIPHER ]; then
echo "PASSED"
PASS=$[PASS + 1]
echo $PASS
fi
done
if [ $PASS == $[LOOP+1] ]; then
echo "All Tests Successful"
fi
And the input file looks like this:
Len = 0
Msg = 00
MD = da39a3ee5e6b4b0d3255bfef95601890afd80709
Len = 1
Msg = 00
MD = bb6b3e18f0115b57925241676f5b1ae88747b08a
Len = 2
Msg = 40
MD = ec6b39952e1a3ec3ab3507185cf756181c84bbe2
All the program does right now, is read the first instances of the variables and loop around there. I'm hoping to use START and END to determine the lines in which it checks the file, and then increment them every time it loops to obtain the other instances of the variable names, but all of my attempts have been unsuccessful so far. Any ideas?
EDIT: Output should look something like, providing my program "./cyassl" works as it should
0
00
da39a3ee5e6b4b0d3255bfef95601890afd80709
da39a3ee5e6b4b0d3255bfef95601890afd80709
PASSED
1
00
bb6b3e18f0115b57925241676f5b1ae88747b08a
bb6b3e18f0115b57925241676f5b1ae88747b08a
PASSED
2
40
ec6b39952e1a3ec3ab3507185cf756181c84bbe2
ec6b39952e1a3ec3ab3507185cf756181c84bbe2
PASSED
etc.
There's no need to make multiple passes on the input file.
#!/bin/sh
exec < ../../Tests/shabittestvectors/SHA1ShortMsg.rsp
status=pass
awk '{print $3,$6,$9}' RS= | {
while read len msg md; do
if test "$(./cyassl hash -sha -i $msg -l $len)" = "$md"; then
echo passed
else
status=fail
fi
done
test "$status" = pass && echo all tests passed
}
The awk will read from stdin (which the exec redirects from the file; personally I would skip that line and have the caller direct input appropriately) and splits the input into records of one paragraph each. A "paragraph" here means that the records are separated by blank lines (the lines must be truly blank, and cannot contain whitespace). Awk then parses each record and prints the 3rd, 6th, and 9th field on a single line. This is a bit fragile, but for the shown input those fields represent length, message, and md hash, respectively. All the awk is doing is rearranging the input so that it is one record per line. Once the data is in a more readable format, a subshell reads the data one line at a time, parsing it into the variables named "len", "msg", and "md". The do loop processes once per line of input, spewing the rather verbose message "passed" with each test it runs (I would remove that, but retained it here for consistency with the original script), and setting the status if any tests fail. The braces are necessary to ensure that the value of the variable status is retained after the do loop terminates.
The following code,
inputfile="../../Tests/shabittestvectors/SHA1ShortMsg.rsp"
while read -r len msg md
do
echo got: LEN:$len MSG:$msg MD:$md
#cypher=$(./cyassl hash -sha -i $msg -l $len)
#continue as you wish
done < <(perl -00 -F'[\s=]+|\n' -lE 'say qq{$F[1] $F[3] $F[5]}' < "$inputfile")
for your input data, produces:
got: LEN:0 MSG:00 MD:da39a3ee5e6b4b0d3255bfef95601890afd80709
got: LEN:1 MSG:00 MD:bb6b3e18f0115b57925241676f5b1ae88747b08a
got: LEN:2 MSG:40 MD:ec6b39952e1a3ec3ab3507185cf756181c84bbe2
If your input data is in order you can have this with a simplified bash:
#!/bin/bash
LOOP=10
PASS=0
FILE='../../Tests/shabittestvectors/SHA1ShortMsg.rsp'
for (( I = 1; I <= LOOP; ++I )); do
read -r LEN && read -r MSG && read -r MD || break
echo "$LEN"
echo "$MSG"
MD=${MD:0:-1}
CIPHER=$(exec ./cyassl hash -sha -i "$MSG" -l "$LEN")
echo "$MD"
echo "$CIPHER"
if [[ $MD == "$CIPHER" ]]; then
echo "PASSED"
(( ++PASS ))
fi
done < <(exec awk '/Len =/,/Msg =/,/MD =/ { print $3 }' "$FILE")
[[ PASS -eq LOOP ]] && echo "All Tests Successful."
Just make sure you don't run it as sh e.g. sh script.sh. bash script.sh most likely.
Here is how I read a file row by row:
while read ROW
do
...
done < file
I don't use the other syntax
cat file | while read ROW
do
...
done
because the pipe creates a subshell and makes me lose the environment variables.
The problem arises if the file doesn't end with a newline: last line is not read. It is easy to solve this in the latter syntax, by echoing just a newline:
(cat file; echo) | while read ROW
do
...
done
How do I do the same in the former syntax, without opening a subshell nor creating a temporary file (the list is quite big)?
A way that works in all shells is the following:
#!/bin/sh
willexit=0
while [ $willexit == 0 ] ; do
read ROW || willexit=1
...
done < file
A direct while read will exit as soon as read encounters the EOF, so the last line will not be processed. By checking the return value outside the while, we can process the last line. An additional test for the emptiness of $ROW should be added after the read though, since otherwise a file whose last line ends with a newline will generate a spurious execution with an empty line, so make it
#!/bin/sh
willexit=0
while [ $willexit == 0 ] ; do
read ROW || willexit=1
if [ -n "$ROW"] ; then
...
fi
done < file
#!/bin/bash
while read ROW
...
done < <(cat file ; echo)
The POSIX way to do this is via a named pipe.
#!/bin/sh
[ -p mypipe ] || mkfifo mypipe
(cat num7; echo) > mypipe &
while read line; do
echo "-->$line<--"
export CNT=$((cnt+1))
done < mypipe
rm mypipe
echo "CNT is '$cnt'"
Input
$ cat infile
1
2
3
4
5$
Output
$ (cat infile;echo) > mypipe & while read line; do echo "-->$line<--"; export CNT=$((cnt+1)); done < mypipe; echo "CNT is '$cnt'"
[1] 22260
-->1<--
-->2<--
-->3<--
-->4<--
-->5<--
CNT is '5'
[1]+ Done ( cat num7; echo ) > mypipe
From an answer to a similar question:
while IFS= read -r LINE || [ -n "${LINE}" ]; do
...
done <file
The IFS= part prevents read from stripping leading and trailing whitespace (see this answer).
If you need to react differently depending on whether the file has a trailing newline or not (e.g., warn the user) you'll have to make some changes to the while condition.