There are 2 pieces of code here, and the value in $1 is the name of a file which contains 3 lines of text.
Now, I have a problem. In the first piece of the code, I can't get the "right" value out of the loop, but in the second piece of the code, I can get the right result. I don't know why.
How can I make the first piece of the code get the right result?
#!/bin/bash
count=0
cat "$1" | while read line
do
count=$[ $count + 1 ]
done
echo "$count line(s) in all."
#-----------------------------------------
count2=0
for var in a b c
do
count2=$[ $count2 + 1 ]
done
echo "$count2 line(s) in all."
This happens because of the pipe before the while loop. It creates a sub-shell, and thus the changes in the variables are not passed to the main script. To overcome this, use process substitution instead:
while read -r line
do
# do some stuff
done < <( some commad)
In version 4.2 or later, you can also set the lastpipe option, and the last command
in the pipeline will run in the current shell, not a subshell.
shopt -s lastpipe
some command | while read -r line; do
# do some stuff
done
In this case, since you are just using the contents of the file, you can use input redirection:
while read -r line
do
# do some stuff
done < "$file"
Related
A script takes a URL, parses it for the required fields, and redirects its output to be saved in a file, file.txt. The output is saved on a new line each time a field has been found.
file.txt
A Cat
A Dog
A Mouse
etc...
I want to take file.txt and create an array from it in a new script, where every line gets to be its own string variable in the array. So far I have tried:
#!/bin/bash
filename=file.txt
declare -a myArray
myArray=(`cat "$filename"`)
for (( i = 0 ; i < 9 ; i++))
do
echo "Element [$i]: ${myArray[$i]}"
done
When I run this script, whitespace results in words getting split and instead of getting
Desired output
Element [0]: A Cat
Element [1]: A Dog
etc...
I end up getting this:
Actual output
Element [0]: A
Element [1]: Cat
Element [2]: A
Element [3]: Dog
etc...
How can I adjust the loop below such that the entire string on each line will correspond one-to-one with each variable in the array?
Use the mapfile command:
mapfile -t myArray < file.txt
The error is using for -- the idiomatic way to loop over lines of a file is:
while IFS= read -r line; do echo ">>$line<<"; done < file.txt
See BashFAQ/005 for more details.
mapfile and readarray (which are synonymous) are available in Bash version 4 and above. If you have an older version of Bash, you can use a loop to read the file into an array:
arr=()
while IFS= read -r line; do
arr+=("$line")
done < file
In case the file has an incomplete (missing newline) last line, you could use this alternative:
arr=()
while IFS= read -r line || [[ "$line" ]]; do
arr+=("$line")
done < file
Related:
Need alternative to readarray/mapfile for script on older version of Bash
You can do this too:
oldIFS="$IFS"
IFS=$'\n' arr=($(<file))
IFS="$oldIFS"
echo "${arr[1]}" # It will print `A Dog`.
Note:
Filename expansion still occurs. For example, if there's a line with a literal * it will expand to all the files in current folder. So use it only if your file is free of this kind of scenario.
Use mapfile or read -a
Always check your code using shellcheck. It will often give you the correct answer. In this case SC2207 covers reading a file that either has space separated or newline separated values into an array.
Don't do this
array=( $(mycommand) )
Files with values separated by newlines
mapfile -t array < <(mycommand)
Files with values separated by spaces
IFS=" " read -r -a array <<< "$(mycommand)"
The shellcheck page will give you the rationale why this is considered best practice.
You can simply read each line from the file and assign it to an array.
#!/bin/bash
i=0
while read line
do
arr[$i]="$line"
i=$((i+1))
done < file.txt
This answer says to use
mapfile -t myArray < file.txt
I made a shim for mapfile if you want to use mapfile on bash < 4.x for whatever reason. It uses the existing mapfile command if you are on bash >= 4.x
Currently, only options -d and -t work. But that should be enough for that command above. I've only tested on macOS. On macOS Sierra 10.12.6, the system bash is 3.2.57(1)-release. So the shim can come in handy. You can also just update your bash with homebrew, build bash yourself, etc.
It uses this technique to set variables up one call stack.
Make sure set the Internal File Separator (IFS)
variable to $'\n' so that it does not put each word
into a new array entry.
#!/bin/bash
# move all 2020 - 2022 movies to /backup/movies
# put list into file 1 line per dir
# dirs are "movie name (year)/"
ls | egrep 202[0-2] > 2020_movies.txt
OLDIFS=${IFS}
IFS=$'\n' #fix separator
declare -a MOVIES # array for dir names
MOVIES=( $( cat "${1}" ) ) // load into array
for M in ${MOVIES[#]} ; do
echo "[${M}]"
if [ -d "${M}" ] ; then # if dir name
mv -v "$M" /backup/movies/
fi
done
IFS=${OLDIFS} # restore standard separators
# not essential as IFS reverts when script ends
#END
A script takes a URL, parses it for the required fields, and redirects its output to be saved in a file, file.txt. The output is saved on a new line each time a field has been found.
file.txt
A Cat
A Dog
A Mouse
etc...
I want to take file.txt and create an array from it in a new script, where every line gets to be its own string variable in the array. So far I have tried:
#!/bin/bash
filename=file.txt
declare -a myArray
myArray=(`cat "$filename"`)
for (( i = 0 ; i < 9 ; i++))
do
echo "Element [$i]: ${myArray[$i]}"
done
When I run this script, whitespace results in words getting split and instead of getting
Desired output
Element [0]: A Cat
Element [1]: A Dog
etc...
I end up getting this:
Actual output
Element [0]: A
Element [1]: Cat
Element [2]: A
Element [3]: Dog
etc...
How can I adjust the loop below such that the entire string on each line will correspond one-to-one with each variable in the array?
Use the mapfile command:
mapfile -t myArray < file.txt
The error is using for -- the idiomatic way to loop over lines of a file is:
while IFS= read -r line; do echo ">>$line<<"; done < file.txt
See BashFAQ/005 for more details.
mapfile and readarray (which are synonymous) are available in Bash version 4 and above. If you have an older version of Bash, you can use a loop to read the file into an array:
arr=()
while IFS= read -r line; do
arr+=("$line")
done < file
In case the file has an incomplete (missing newline) last line, you could use this alternative:
arr=()
while IFS= read -r line || [[ "$line" ]]; do
arr+=("$line")
done < file
Related:
Need alternative to readarray/mapfile for script on older version of Bash
You can do this too:
oldIFS="$IFS"
IFS=$'\n' arr=($(<file))
IFS="$oldIFS"
echo "${arr[1]}" # It will print `A Dog`.
Note:
Filename expansion still occurs. For example, if there's a line with a literal * it will expand to all the files in current folder. So use it only if your file is free of this kind of scenario.
Use mapfile or read -a
Always check your code using shellcheck. It will often give you the correct answer. In this case SC2207 covers reading a file that either has space separated or newline separated values into an array.
Don't do this
array=( $(mycommand) )
Files with values separated by newlines
mapfile -t array < <(mycommand)
Files with values separated by spaces
IFS=" " read -r -a array <<< "$(mycommand)"
The shellcheck page will give you the rationale why this is considered best practice.
You can simply read each line from the file and assign it to an array.
#!/bin/bash
i=0
while read line
do
arr[$i]="$line"
i=$((i+1))
done < file.txt
This answer says to use
mapfile -t myArray < file.txt
I made a shim for mapfile if you want to use mapfile on bash < 4.x for whatever reason. It uses the existing mapfile command if you are on bash >= 4.x
Currently, only options -d and -t work. But that should be enough for that command above. I've only tested on macOS. On macOS Sierra 10.12.6, the system bash is 3.2.57(1)-release. So the shim can come in handy. You can also just update your bash with homebrew, build bash yourself, etc.
It uses this technique to set variables up one call stack.
Make sure set the Internal File Separator (IFS)
variable to $'\n' so that it does not put each word
into a new array entry.
#!/bin/bash
# move all 2020 - 2022 movies to /backup/movies
# put list into file 1 line per dir
# dirs are "movie name (year)/"
ls | egrep 202[0-2] > 2020_movies.txt
OLDIFS=${IFS}
IFS=$'\n' #fix separator
declare -a MOVIES # array for dir names
MOVIES=( $( cat "${1}" ) ) // load into array
for M in ${MOVIES[#]} ; do
echo "[${M}]"
if [ -d "${M}" ] ; then # if dir name
mv -v "$M" /backup/movies/
fi
done
IFS=${OLDIFS} # restore standard separators
# not essential as IFS reverts when script ends
#END
I have a Bash script where I want to count how many things were done when looping through a file. The count seems to work within the loop but after it the variable seems reset.
nKeys=0
cat afile | while read -r line
do
#...do stuff
let nKeys=nKeys+1
# this will print 1,2,..., etc as expected
echo Done entry $nKeys
done
# PROBLEM: this always prints "... 0 keys"
echo Finished writing $destFile, $nKeys keys
The output of the above is something alone the lines of:
Done entry 1
Done entry 2
Finished writing /blah, 0 keys
The output I want is:
Done entry 1
Done entry 2
Finished writing /blah, 2 keys
I am not quite sure why nKeys is 0 after the loop :( I assume it's something basic but damned if I can spot it despite looking at http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-7.html and other resources.
Fingers crossed someone else can look at it and go "well duh! You have to ..."!
In the just-released Bash 4.2, you can do this to prevent creating a subshell:
shopt -s lastpipe
Also, as you'll probably see at the link Ignacio provided, you have a Useless Use of cat.
while read -r line
do
...
done < afile
As mentioned in the accepted answer, this happens because pipes spawn separate subprocesses. To avoid this, command grouping has been the best option for me. That is, doing everything after the pipe in a subshell.
nKeys=0
cat afile |
{
while read -r line
do
#...do stuff
let nKeys=nKeys+1
# this will print 1,2,..., etc as expected
echo Done entry $nKeys
done
# PROBLEM: this always prints "... 0 keys"
echo Finished writing $destFile, $nKeys keys
}
Now it will report the value of $nKeys "correctly" (i.e. what you wish).
I arrived at the desired result in the following way without using pipes or here documents
#!/bin/sh
counter=0
string="apple orange mango egg indian"
str_len=${#string}
while [ $str_len -ne 0 ]
do
c=${string:0:1}
if [[ "$c" = [aeiou] ]]
then
echo -n "vowel : "
echo "- $c"
counter=$(( $counter + 1 ))
fi
string=${string:1}
str_len=${#string}
done
printf "The number of vowels in the given string are : %s "$counter
echo
I intend to read the lines of a short .txt file and assign each line to variables containing the line number in the variable name.
File example.txt looks like this:
Line A
Line B
When I run the following code:
i=1
while read line; do
eval line$i="$line"
echo $line
((i=i+1))
done < example.txt
What I would expect during execution is:
Line A
Line B
and afterwards being able to call
$ echo $line1
Line A
$ echo $line2
Line B
However, the code above results in the error:
-bash: A: command not found
Any ideas for a fix?
Quote-removal happens twice with eval. Your double-quotes are getting removed before eval even runs. I'm not even going to directly answer that part, because there are better ways to do this:
readarray line < example.txt # bash 4
echo "${line[0]}"
Or, to do exactly what you were doing, with a different variable for each line:
i=1
while read line$((i++)); do
:
done < example.txt
Also check out printf -v varname "%s" value for a better / safer way to assign by reference.
Check out the bash-completion code if you want to see some complicated call-by-reference bash shenanigans.
Addressing your comment: if you want to process lines as they come in, but still save previous lines, I'd go with this construct:
lines=()
while read l;do
lines+=( "$l" )
echo "my line is $l"
done < "$infile"
This way you don't have to jump through any syntactic hoops to access the current line (vs. having to declare a reference-variable to line$i, or something.)
Bash arrays are really handy, because you can access a single element by value, or you can do "${#lines[#]}" to get the line count. Beware that unset lines[4] leaves a gap, rather than renumbering lines[5:infinity]. See the "arrays" section in the bash man page. To find the part of the manual that documents $# expansion, and other stuff, search in the manual for ##. The Parameter Expansion section is the first hit for that in the bash 4.3 man page.
eval line$i="$line" tells bash to evaluate the string "line1=Line A", which attempts to invoke a command named A with the environment variable "line1" set to the value of Line. You probably want to do eval "line$i='$line'"
I am trying to write a function in a bash script that gets lines from stdin and picks out the first line which is not contained in a file.
Here is my approach:
doubles=file.txt
firstnotdouble(){
while read input_line; do
found=0;
cat $doubles |
while read double_line; do
if [ "$input_line" = "$double_line" ]
then
found=1;
break
fi
done
if [ $found -eq 0 ] # no double found, echo and break!
then
echo $input_line
break
fi
done
}
After some debugging attempts I realized that when found is set to 1 in the first if block, it does not keep its value until the next if block. That's why it's not working. Why does the script act as if there were two found variables in different "scopes"?
The second question would be if the approach as a whole could be optimized.
As indicated in the comments, the issue with environment variables is that the commands in a pipeline (that is, a series of commands separated by |) run in subshells, and each subshell has its own environment variables. You could have avoided the problem by avoiding the UUOC (useless use of cat), writing:
while read ...; do ... done < "$doubles"
instead of the pipeline.
A (much) faster way than using a while read loop repeatedly through the doubles file is to use grep:
# Specify the file to be scanned as the first argument
firstnotdouble() {
while IFS= read -r double_line; do
if ! grep -qxF "$double_line" "$1"; then
echo "$double_line"
return
fi
done
return 1
}
In the grep:
-q suppress print out, and stop on first match
-x pattern must match the entire line
-F pattern is a simple string instead of a regular expression.
In the read:
IFS= avoids spaces being trimmed
-r avoids backslashes being deleted
With GNU grep, you could use -xF -m1 (or even -xFm1 if you like being cryptic) instead of -qxF, and then leave out the echo. The grep extension -m N limits the number of matches found to N.