Syntax error with use of pipe in bash - bash

I'm diving headfirst into bash with no prior experience and have hit a bit of a snag: I wrote a small bash script to determine the average of values (it's just a total right now) being returned by a c executable.
#Sample value of #s1total: 0+0.000117+0.000149+0.000106
printf "\n%s" $s1total
#The following line works
printf "\nTotal: %s\n" $(bc <<< $s1total)
#the following also works
echo
echo -n "Total: "
echo $s1total | bc
#The following line does not work
printf "\nTotal: %s\n" $($s1total|bc)
I did eventually find that the last line can be made to work by changing it to $(echo $s1total|bc) but i don't understand why that works ...
If I run as-is, I get an error:./sievetest.sh: line 25: 0+0.000117+0.000149+0.000106: command not found
Is the string being run before the pipe? Why? Why does adding "echo" fix it?
How is the third method different than the first and second?
(as an aside, I thought the heredoc redirection operator was << , why the extra < ?)

You have this:
$($s1total|bc)
which is basically
$(0+0.000117+0.000149+0.000106 | bc)
e.g. run a command named 0+.... and pipe it to bc.
It should be
$(echo $s1total|bc)

Related

Bash: Loop through lines of a file and assign line to numbered variable names

I intend to read the lines of a short .txt file and assign each line to variables containing the line number in the variable name.
File example.txt looks like this:
Line A
Line B
When I run the following code:
i=1
while read line; do
eval line$i="$line"
echo $line
((i=i+1))
done < example.txt
What I would expect during execution is:
Line A
Line B
and afterwards being able to call
$ echo $line1
Line A
$ echo $line2
Line B
However, the code above results in the error:
-bash: A: command not found
Any ideas for a fix?
Quote-removal happens twice with eval. Your double-quotes are getting removed before eval even runs. I'm not even going to directly answer that part, because there are better ways to do this:
readarray line < example.txt # bash 4
echo "${line[0]}"
Or, to do exactly what you were doing, with a different variable for each line:
i=1
while read line$((i++)); do
:
done < example.txt
Also check out printf -v varname "%s" value for a better / safer way to assign by reference.
Check out the bash-completion code if you want to see some complicated call-by-reference bash shenanigans.
Addressing your comment: if you want to process lines as they come in, but still save previous lines, I'd go with this construct:
lines=()
while read l;do
lines+=( "$l" )
echo "my line is $l"
done < "$infile"
This way you don't have to jump through any syntactic hoops to access the current line (vs. having to declare a reference-variable to line$i, or something.)
Bash arrays are really handy, because you can access a single element by value, or you can do "${#lines[#]}" to get the line count. Beware that unset lines[4] leaves a gap, rather than renumbering lines[5:infinity]. See the "arrays" section in the bash man page. To find the part of the manual that documents $# expansion, and other stuff, search in the manual for ##. The Parameter Expansion section is the first hit for that in the bash 4.3 man page.
eval line$i="$line" tells bash to evaluate the string "line1=Line A", which attempts to invoke a command named A with the environment variable "line1" set to the value of Line. You probably want to do eval "line$i='$line'"

Using a BASH script variable into an executed command

Can someone please help with this because I can't seem to find a solution. I have the following script that works fine:
#!/bin/bash
#Checks the number of lines in the userdomains file
NUM=`awk 'END {print NR}' /etc/userdomains.hristian`;
echo $NUM
#Prints out a particular line from the file (should work with $NUM eventually)
USER=`sed -n 4p /etc/userdomains.hristian`
echo $USER
#Edits the output so that only the username is left
USER2=`echo $USER | awk '{print $NF}'`
echo $USER2
However, when I substitute the 4 on line 12 with the variable $NUM, like this, it doesn't work:
USER=`sed -n $NUMp /etc/userdomains.hristian`
I tried a number of different combinations of quotes and ${}, however none of them seem to work because I'm a BASH newbie. Help please :)
I'm not sure exactly what you've already tried but this works for me:
$ cat out
line 1
line 2
line 3
line 4
line 5
$ num=4
$ a=`sed -n ${num}p out`
$ echo "$a"
line 4
To be clear, the issue here is that you need to separate the expansion of $num from the p in the sed command. That's what the curly braces do.
Note that I'm using lowercase variable names. Uppercase ones should be be reserved for use by the shell. I would also recommend using the more modern $() syntax for command substitution:
a=$(sed -n "${num}p" out)
The double quotes around the sed command aren't necessary but they don't do any harm. In general, it's a good idea to use them around expansions.
Presumably the script in your question is a learning exercise, which is why you have done all of the steps separately. For the record, you could do the whole thing in one go like this:
awk 'END { print $NF }' /etc/userdomains.hristian
In the END block, the values from the last line in the file can still be accessed, so you can print the last field directly.
Your trying to evaluate the variable $NUMp rather than $NUM. Try this instead:
USER=`sed -n ${NUM}p /etc/userdomains.hristian`

Read last line of file in bash script when reading file line by line [duplicate]

This question already has answers here:
Read a file line by line assigning the value to a variable [duplicate]
(10 answers)
Closed 3 years ago.
I am writing a script to read commands from a file and execute a specific command. I want my script to work for either single input arguments or when an argument is a filename which contains the arguments in question.
My code below works except for one problem, it ignores the last line of the file. So, if the file were as follows.
file.txt
file1
file2
The script posted below only runs the command for file.txt
for currentJob in "$#"
do
if [[ "$currentJob" != *.* ]] #single file input arg
then
echo $currentJob
serverJobName="$( tr '[A-Z]' '[a-z]' <<< "$currentJob" )" #Change to lowercase
#run cURL job
curl -o "$currentJob"PisaInterfaces.xml http://www.ebi.ac.uk/msd-srv/pisa/cgi-bin/interfaces.pisa?"$serverJobName"
else #file with list of pdbs
echo "Reading "$currentJob
while read line; do
echo "-"$line
serverJobName="$( tr '[A-Z]' '[a-z]' <<< "$line" )"
curl -o "$line"PisaInterfaces.xml http://www.ebi.ac.uk/msd-srv/pisa/cgi-bin/interfaces.pisa?"$serverJobName"
done < "$currentJob"
fi
done
There is, of course, the obvious work around where after the while loop I repeat the steps for inside the loop to complete those commands with the last file, but this is not desirable as any changes I make inside the while loop must be repeated again outside of the while loop. I have searched around online and could not find anyone asking this precise question. I am sure it is out there, but I have not found it.
The output I get is as follows.
>testScript.sh file.txt
Reading file.txt
-file1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 642k 0 642k 0 0 349k 0 --:--:-- 0:00:01 --:--:-- 492k
My bash version is 3.2.48
It sounds like your input file is missing the newline character after its last line. When read encounters this, it sets the variable ($line in this case) to the last line, but then returns an end-of-file error so the loop doesn't execute for that last line. To work around this, you can make the loop execute if read succeeds or it read anything into $line:
...
while read line || [[ -n "$line" ]]; do
...
EDIT: the || in the while condition is what's known as a short-circuit boolean -- it tries the first command (read line), and if that succeeds it skips the second ([[ -n "$line" ]]) and goes through the loop (basically, as long as the read succeeds, it runs just like your original script). If the read fails, it checks the second command ([[ -n "$line" ]]) -- if read read anything into $line before hitting the end of file (i.e. if there was an unterminated last line in the file), this'll succeed, so the while condition as a whole is considered to have succeeded, and the loop runs one more time.
After that last unterminated line is processed, it'll run the test again. This time, the read will fail (it's still at the end of file), and since read didn't read anything into $line this time the [[ -n "$line" ]] test will also fail, so the while condition as a whole fails and the loop terminates.
EDIT2: The [[ ]] is a bash conditional expression -- it's not a regular command, but it can be used in place of one. Its primary purpose is to succeed or fail, based on the condition inside it. In this case, the -n test means succeed if the operand ("$line") is NONempty. There's a list of other test conditions here, as well as in the man page for the test command.
Note that a conditional expression in [[ ... ]] is subtly different from a test command [ ... ] -- see BashFAQ #031 for differences and a list of available tests. And they're both different from an arithmetic expression with (( ... )), and completely different from a subshell with( ... )...
Your problem seems to be a missing carriage return in your file.
If you cat your file, you need to see the last line successfully appearing before the promopt.
Otherwise try adding :
echo "Reading "$currentJob
echo >> $currentJob #add new line
while read line; do
to force the last line of the file.
Using grep with while loop:
while IFS= read -r line; do
echo "$line"
done < <(grep "" file)
Using grep . instead of grep "" will skip the empty lines.
Note:
Using IFS= keeps any line indentation intact.
You should almost always use the -r option with read.
I found some code that reads a file including the last line, and works without the [[ ]] command:
http://www.unix.com/shell-programming-and-scripting/161645-read-file-using-while-loop-not-reading-last-line.html
DONE=false
until $DONE; do
read s || DONE=true
# you code
done < FILE

Respect last line if it's not terminated with a new line char (\n) when using read

I have noticed for a while that read never actually reads the last line of a file if there is not, at the end of it, a "newline" character. This is understandable if one consider that, as long as there is not a "newline" character in a file, it is as if it contained 0 line (which is quite difficult to admit !). See, for example, the following:
$ echo 'foo' > bar ; wc -l bar
1 bar
But...
$ echo -n 'bar' > foo ; wc -l foo
0 foo
The question is then: how can I handle such situations when using read to process files which have not been created or modified by myself, and about which I don't know if they actually end up with a "newline" character ?
read does, in fact, read an unterminated line into the assigned var ($REPLY by default). It also returns false on such a line, which just means ‘end of file’; directly using its return value in the classic while loop thus skips that one last line. If you change the loop logic slightly, you can process non-new line terminated files correctly, without need for prior sanitisation, with read:
while read -r || [[ -n "$REPLY" ]]; do
# your processing of $REPLY here
done < "/path/to/file"
Note this is much faster than solutions relying on externals.
Hat tip to Gordon Davisson for improving the loop logic.
POSIX requires any line in a file have a newline character at the end to denote it is a line. But this site offers a solution to exactly the scenario you are describing. Final product is this chunklet.
newline='
'
lastline=$(tail -n 1 file; echo x); lastline=${lastline%x}
[ "${lastline#"${lastline%?}"}" != "$newline" ] && echo >> file
# Now file is sane; do our normal processing here...
If you must use read, try this:
awk '{ print $0}' foo | while read line; do
echo the line is $line
done
as awk seems to recognize lines even without the newline char
This is more or less a combination of the answers given so far.
It does not modify the files in place.
(cat file; tail -c1 file | grep -qx . && echo) | while read line
do
...
done

How to use the read command in Bash?

When I try to use the read command in Bash like this:
echo hello | read str
echo $str
Nothing echoed, while I think str should contain the string hello. Can anybody please help me understand this behavior?
The read in your script command is fine. However, you execute it in the pipeline, which means it is in a subshell, therefore, the variables it reads to are not visible in the parent shell. You can either
move the rest of the script in the subshell, too:
echo hello | { read str
echo $str
}
or use command substitution to get the value of the variable out of the subshell
str=$(echo hello)
echo $str
or a slightly more complicated example (Grabbing the 2nd element of ls)
str=$(ls | { read a; read a; echo $a; })
echo $str
Other bash alternatives that do not involve a subshell:
read str <<END # here-doc
hello
END
read str <<< "hello" # here-string
read str < <(echo hello) # process substitution
Typical usage might look like:
i=0
echo -e "hello1\nhello2\nhello3" | while read str ; do
echo "$((++i)): $str"
done
and output
1: hello1
2: hello2
3: hello3
The value disappears since the read command is run in a separate subshell: Bash FAQ 24
To put my two cents here: on KSH, reading as is to a variable will work, because according to the IBM AIX documentation, KSH's read does affects the current shell environment:
The setting of shell variables by the read command affects the current shell execution environment.
This just resulted in me spending a good few minutes figuring out why a one-liner ending with read that I've used a zillion times before on AIX didn't work on Linux... it's because KSH does saves to the current environment and BASH doesn't!
I really only use read with "while" and a do loop:
echo "This is NOT a test." | while read -r a b c theRest; do
echo "$a" "$b" "$theRest"; done
This is a test.
For what it's worth, I have seen the recommendation to always use -r with the read command in bash.
You don't need echo to use read
read -p "Guess a Number" NUMBER
Another alternative altogether is to use the printf function.
printf -v str 'hello'
Moreover, this construct, combined with the use of single quotes where appropriate, helps to avoid the multi-escape problems of subshells and other forms of interpolative quoting.
Do you need the pipe?
echo -ne "$MENU"
read NUMBER

Resources