While-loop over lines from variable in bash - bash

Assume a file file with multiple lines.
$ cat file
foo
bar
baz
Assume further that I wish to loop through each line with a while-loop.
$ while IFS= read -r line; do
$ echo $line
$ # do stuff
$ done < file
foo
bar
baz
Finally, please assume that I wish to pass lines stored in a variable rather than lines stored in a file. How can I loop through lines that are saved as a variable without receiving the below error?
$ MY_VAR=$(cat file)
$ while IFS= read -r line; do
$ echo $line
$ # do stuff
$ done < $(echo "$MY_VAR")
bash: $(echo "$MY_VAR"): ambiguous redirect

You have several options:
A herestring (note that this is a non-POSIX extension): done <<<"$MY_VAR"
A heredoc (POSIX-compliant, will work with /bin/sh):
done <<EOF
$MY_VAR
EOF
A process substitution (also a non-POSIX extension, but using printf rather than echo makes it more predictable across shells that support it; see the APPLICATION USAGE note in the POSIX spec for echo): done < <(printf '%s\n' "$MY_VAR")
Note that the first two options will (in bash) create a temporary file on disk with the variable's contents, whereas the last one uses a FIFO.

< needs to be followed by a filename. You can use a here-string:
done <<< "$MY_VAR"
or process substitution:
done < <(echo "$MY_VAR")

Related

This loop will only ever run once. Bad quoting or missing glob/expansion?

Im working on a bash script that recive a list of processes and do a bunch of things with them, however when I want to analyze them with a loop this error happens.
Here is some code:
#! /bin/bash
ls /proc | grep ^9 > processes.txt
cat processes.txt
for line in $processes.txt
do
echo "$line"
done
PD: Im preatty new to bash
$ does parameter expansion; it does not expand a file name to the contents of the file.
Use a while read loop instead.
while IFS= read -r line; do
echo "$line"
done < processes.txt

Why does splitting my $PATH with `read -r -a line` work but not with `while read -r line`?

Just noticed something strange which I can't quite explain:
When I split my $PATH variable using read -a everything works fine
IFS=: read -r -a lines <<< "$PATH"
for line in "${lines[#]}"; do echo "$line"; done
But when I try to do the same using while ... read loop, only the first line is printed
while IFS=: read -r line; do echo "$line"; done <<< "$PATH"
You can make this work; switch from using IFS=: to using -d:, and append a : to the end of your input stream:
while IFS= read -r -d: line; do echo "$line"; done <<< "$PATH:"
The difference is that IFS is used to find boundaries between words, but read -r line reads into exactly one variable, line, so it's not looking for multiple words at all. By contrast, -d tells each invocation of read which character to stop at; by default that's a newline, but you can replace it with any other single character. (If that character isn't found, read exits with a nonzero status; that's why the standard/idiomatic while read loop idiom skips the last line of your file if it isn't correctly terminated by a newline, and why we use $PATH: as our input here).
If you ran IFS=: read -r first second rest, on the other hand, it would put your first PATH entry into $first, the second one into $second, and the remainder of the line into $rest; whereas with IFS: read -r line, it's as if you only had a single item, $rest.
Your while loop processes 1 line, it is not a loop. So the complete path is stored in the field line.
When you had given more fields, the path would be divided to those fields (and the last field gets the remainder):
while IFS=: read -r line field2 field3 otherfields; do echo "$line"; done <<< "$PATH"
When you want to avoid an array, you can use
while read -r line; do echo "$line"; done <<< "${PATH//:/$'\n'}"
It works fine.
Splitting into an array gives an open-ended number of elements, so does what you expect.
Splitting into a single variable does the same thing, but when it runs out of supplied variable names into which to put the data, it's stops splitting and puts the rest into the last one.
Try this:
$: IFS=: read -r a b c <<< "$PATH"
$: printf "[%s]\n" "$a" "$b" "$c"
You'll get the first PATH element in $a, the second in $b, and the rest ALL in $c.
Does that make it clearer?
c.f. this guide
Why does splitting my $PATH with read -r -a line work but not with while read -r line?
Because read -r line reads the whole line and then after reading the single whole line then the line is spitt on IFS. Because you provided only one variable to read, all the line is in that one variable. You could like split the line on the first element and rest of elements:
IFS=: read -r part1 rest_of_parts <<<"$line"
See read 1p read the If there are fewer vars than fields, part. Note that still IFS=: read -r -a lines <<< "$PATH" will fail when PATH contains a newline, like so:
$ export PATH=/usr/bin # reset PATH to something short
$ cd /tmp/
$ mkdir temp$'\n'dir # create a directory with a newline in the name
$ ls -d tem*
'temp'$'\n''dir'/
$ cd temp$'\n'dir
$ printf "%s\n" '#!/bin/bash' 'echo hello world' > script.sh
$ chmod +x ./script.sh # add a script in that directory
$ export PATH="$PATH:$PWD" # add that directory to path
$ ./script.sh # yes. yes, it works
hello world
$ IFS=: read -r -a lines <<< "$PATH"
$ declare -p lines
declare -a lines=([0]="/usr/bin" [1]="/tmp/temp")
# ^^^^ newline and 'dir' is missing
# That is because `read` reads _one line_ and one line only
# _after_ reading that one line that _one line_ is split on IFS
# so any more lines are ignored.
You could use a bash extension to read -d that makes read not read the whole line, but up until a character (but I needed to ignore read exit status, dunno why):
$ while IFS= read -r -d':' line || [[ -n "$line" ]]; do declare -p line; done < <(printf "%s" "$PATH")
declare -- line="/usr/bin"
declare -- line="/tmp/temp
dir"
Note that <<< adds a trailing newline, so using that will result in the last element of PATH having a newline - as a workaround, in bash you may use process substitution < <(printf "%s" "$PATH").
The real safe solution if using bash is just using mapfile/readarray:
$ mapfile -d: -t lines < <(printf "%s" "$PATH")
$ declare -p lines
declare -a lines=([0]="/usr/bin" [1]=$'/tmp/temp\ndir')

Shell script for replacing characters?

I'm trying to write a shell script that takes in a file(ex. file_1_2.txt) and replaces any "_" with "."(ex. file.1.2.txt). This is what I have but its giving me a blank output when I run it.
read $var
x= `echo $var | sed 's/\./_/g'`
echo $x
I'm trying to store the changed filename in the variable "x" and then output x to the console.
I am calling this script by writing
./script2.sh < file_1_2.txt
There is two problems. First, your code has some bugs:
read var
x=`echo $var | sed 's/_/\./g'`
echo $x
will work. You had an extra $ in read var, a space too much (as mentioned before) and you mixed up the replacement pattern in sed (it was doing the reverse of what you wanted).
Also if you want to replace the _ by . in the filename you should do
echo "file_1_2.txt" | ./script2.sh
If you use < this will read the content of `file_1_2.txt" into your script.
Another solution, with bash only:
$ x=file_1_2.txt; echo "${x//_/.}"
file.1.2.txt
(See “Parameter expansion” section in bash manual page for details)
And you can also do this with rename:
$ touch file_1_2.txt
$ ls file*
file_1_2.txt
$ rename 'y/_/\./' file_1_2.txt
$ ls file*
file.1.2.txt
Threre is not need for sed as bash supports variable replacement:
$ cat ./script2
#!/bin/bash
ofile=$1
nfile=${ofile//_/./}
echo mv "$ofile" "$nfile"
$ ./script2 file_1_2.txt
mv "file_1_2.txt" "file.1.2.txt"
Then just remove echo if you are satisfied with the result.

Shell POSIX two nested while read and read from stdin not working

I have that sample script:
#!/bin/sh
while read ll </dev/fd/4; do
echo "1 "$ll
while read line; do
echo $line
read input </dev/fd/3
echo "$input"
done 3<&0 <notify-finished
done 4<output_file
Currently The first loop do not iterate just stays on line 1. How do I fix that without bashisms because it has to be highly portable. Thanks.
Your code already has bashisms. Here, I'm taking them out (and simplifying the FD handling for better readability):
#!/bin/sh
while read ll <&4; do # read from output_file
printf '%s\n' "1 $ll"
while read line <&3; do # read from notify-finished
printf '%s\n' "$line"
read input # read from stdin
printf '%s\n' "$input"
done 3<notify-finished
done 4<output_file
Run the script as follows:
echo "output_file" >output_file
echo "notify-finished" >notify-finished
echo "stdout" | ./yourscript
...and it correctly exits with the following output:
1 output_file
notify-finished
stdout
Notes:
echo's behavior is wildly nonportable across POSIX platforms. See the APPLICATION USAGE section of the POSIX spec for echo, which advises using printf instead.
/dev/fd/## is not specified by POSIX; it is an extension made available both by Linux distributions (creating a symlink to /proc/self/fd -- /proc being itself an unspecified extension) and by bash itself. Use <&4 in place of </dev/fd/4.
You probably want to use the -r argument to read -- which is POSIX-specified, and prevents the default behavior of treating backslashes as escape sequences for newlines and characters in IFS. Without it, foo\bar is read as foobar, thus not reading your data as it truly exists in its input sources.

Is there any difference in bash between "while read -r line; do ...; done < file` and `cat file | while IFS= read -r line; do ...; done`?

I'm learning bash and I found a tutorial on internet that says these are the same:
while read -r line;
do
...
done < file
$ cat file | while IFS= read -r line;
do
...
done
Are there any subtle differences in these two loops are are they really the same?
The biggest difference is that in the pipeline, the while loop executes in a subshell, so if you change the values of any variables in the body of the while, those will be lost after the pipeline completes.
$ foo=5
$ cat file | while IFS= read -r line; do
> foo=$line # assume $line is not 5
> done
$ echo $foo
5
$ while IFS= read -r line; do
> foo=$line
> done < file # Assume one line with the word foo
$ echo $foo
foo
In bash 4.2, this can be mitigated by using the lastpipe option, which allows the last command in a pipeline to be executed in the current shell instead of a subshell.
Aside from that, the version using input redirection is more efficient, since it does not require extra processes to be started.
In addition to chepner's observation about subshells, one of the loops uses IFS= and one does not.
read uses this variable to split up words. With one variable, this affects leading and trailing whitespace.
With IFS=, it's preserved:
$ IFS= read -r line <<< " test "
$ printf "<%s>\n" "$line"
< test >
Otherwise, it's stripped:
$ read -r line <<< " test "
$ printf "<%s>\n" "$line"
<test>
You can imagine how much havoc the first non-IFS= loop would wreck on e.g. a Python file.

Resources