Here is how I read a file row by row:
while read ROW
do
...
done < file
I don't use the other syntax
cat file | while read ROW
do
...
done
because the pipe creates a subshell and makes me lose the environment variables.
The problem arises if the file doesn't end with a newline: last line is not read. It is easy to solve this in the latter syntax, by echoing just a newline:
(cat file; echo) | while read ROW
do
...
done
How do I do the same in the former syntax, without opening a subshell nor creating a temporary file (the list is quite big)?
A way that works in all shells is the following:
#!/bin/sh
willexit=0
while [ $willexit == 0 ] ; do
read ROW || willexit=1
...
done < file
A direct while read will exit as soon as read encounters the EOF, so the last line will not be processed. By checking the return value outside the while, we can process the last line. An additional test for the emptiness of $ROW should be added after the read though, since otherwise a file whose last line ends with a newline will generate a spurious execution with an empty line, so make it
#!/bin/sh
willexit=0
while [ $willexit == 0 ] ; do
read ROW || willexit=1
if [ -n "$ROW"] ; then
...
fi
done < file
#!/bin/bash
while read ROW
...
done < <(cat file ; echo)
The POSIX way to do this is via a named pipe.
#!/bin/sh
[ -p mypipe ] || mkfifo mypipe
(cat num7; echo) > mypipe &
while read line; do
echo "-->$line<--"
export CNT=$((cnt+1))
done < mypipe
rm mypipe
echo "CNT is '$cnt'"
Input
$ cat infile
1
2
3
4
5$
Output
$ (cat infile;echo) > mypipe & while read line; do echo "-->$line<--"; export CNT=$((cnt+1)); done < mypipe; echo "CNT is '$cnt'"
[1] 22260
-->1<--
-->2<--
-->3<--
-->4<--
-->5<--
CNT is '5'
[1]+ Done ( cat num7; echo ) > mypipe
From an answer to a similar question:
while IFS= read -r LINE || [ -n "${LINE}" ]; do
...
done <file
The IFS= part prevents read from stripping leading and trailing whitespace (see this answer).
If you need to react differently depending on whether the file has a trailing newline or not (e.g., warn the user) you'll have to make some changes to the while condition.
Related
I need an output of current count vs total count in single line. I would like to know if this could be done Via Bash using 'for' 'while' loop.
Expecting an output that should only update the count and should not display multiple lines
File Content
$ cat ~/test.rtf
hostname1
hostname2
hostname3
hostname4
#!/bin/sh
j=1
k=$(cat ~/test.rtf | wc -l)
for i in $(cat ~/test.rtf);
do
echo "Working on line ($j/$k)"
echo "$i"
#or any other command for i
j=$((j+1))
done
EX:
Working on line (2/4)
Not like,
Working on line (2/4)
Working on line (3/4)
Assumptions:
OP wants to generate n lines of output that overwrite each other on successive passes through the loop
in OP's sample code there are two echo calls so in this case n=2
General approaches:
issue a clear at the beginning of each pass through the loop so as to clear the current output and reposition the cursor at the 'top' of the console/window
use tput to manage movement of the cursor (and clearing/overwriting of previous output)
Sample input:
$ cat test.rtf
this is line1
then line2
and line3
and last but not least line4
real last line5
clear approach:
j=1
k=$(wc -l < test.rtf)
while read -r line
do
clear
echo "Working on line ($j/$k)"
echo "${line}"
((j++))
done < test.rtf
tput approach:
j=1
k=$(wc -l < test.rtf)
EraseToEOL=$(tput el) # grab terminal specific code for clearing from cursor to EOL
clear # optional: start with a new screen otherwise use current position in console/window for next command ...
tput sc # grab current cursor position
while read -r line
do
tput rc # go (back) to cursor position stored via 'tput sc'
echo "Working on line ($j/$k)"
echo "${line}${EraseToEOL}" # ${EraseToEOL} forces clearing rest of line, effectively erasing a previous line that may have been longer then the current line of output
((j++))
done < test.rtf
Both of these generate the same result:
Something along these lines:
file=~/test.rtf
nl=$(wc -l "$file")
nl=${nl%%[[:blank:]]*}
i=0
while IFS= read -r line; do
i=$((i+1))
echo "Working on line ($i/$nl)"
done < "$file"
Your main question is how to avoid each the counter to be written to new lines. The newlines are \n characters, which is appended by echo. You want \r, like
for ((i=0; i<10; i++)); do
printf "Counter $i\r"
sleep 1
done
echo
When you echo something from the line you are working on, you will use \n again. I will use cut as an example of processing the inputline. Use the output string in the same printf command like
j=1
k=$(cat ~/test.rtf | wc -l)
while IFS= read -r line; do
printf "Working on line (%s): %s\r" "$j/$k" $(cut -c1-10 <<< "${line}")
sleep 1
((j++))
done < ~/test.rft
The problem with the above solution is that you will see output from previous lines when your last output is shorter than the previous one. When you know the maximum length that your processing of the line will show, you can force additional spaces:
j=1
k=$(cat ~/test.rtf | wc -l)
while IFS= read -r line; do
printf "Working on line (%5.5s): %-20s\r" "$j/$k" "$(cut -c1-20 <<< "${line}")";
sleep 1
((j++))
done < ~/test.rft
I have a series of commands chained together with pipes:
should_create_one_line | expects_one_line
The first command should_create_one_line should produce an output that only has one line, but under strange circumstances it is possible for the output to be multiline or empty.
I would like to add a step in between these two, validate_one_line:
should_create_one_line | validate_one_line | expects_one_line
If its input contains exactly 1 line then validate_one_line will simply output its input. If its input contains more than 1 line or is empty then validate_one_line should cause the whole sequence of steps to stop and return an error code.
What command can I use for validate_one_line?
Use read. Here's a shell function that meets your specs:
exactly_one_line() {
local line # Use to echo the line
read -r line || return # Guarantee at least one line is read
read && return 1 # Indicate failure if another line is successfully read
echo "$line"
}
Notes
"One line" assumes a single line followed by a newline. If your input could be like, a file with contents but no newlines, then this will fail.
Given a pipeline like a|b, a cannot prevent b from running. At a minimum, b needs to handle when a produces no output.
Demo:
$ wc -l empty oneline twolines
0 empty
1 oneline
2 twolines
3 total
$ exactly_one_line < empty; echo $?
1
$ exactly_one_line < oneline; echo $?
oneline
0
$ exactly_one_line < twolines; echo $?
1
First off, you should seriously consider adding the validation code to expects_one_line. According to this post, each process starts in its own subshell, meaning that even if validate_one_line fails, you will get an error in expects_one_line because it will try to run with no input (or a blank line). That being said, here is a bash one-liner that you can insert into your pipe to validate:
should_create_one_line.sh | ( var="$(cat)"; [ $(echo "$var" | wc -l) -ne 1 ] && exit 1 || echo "$var") | expects_one_line.sh
The problem here is that when the validation subshell returns in the exit 1 case, expects_one_line.sh will still get a single blank line. If this works for you, then great. If not, it would be better to just put the following into the beginning of expects_one_line.sh:
input="$(cat)"
[ $(echo "$var" | wc -l) -ne 1 ] && exit 1
This would guarantee that expects_one_line.sh fails properly when getting a single line without having to wonder about what the empty line that the validation outputs will do to the script.
You may find this post helpful: How to read mutliline input from stdin into variable and how to print one out in shell(sh,bash)?
You can use a bash script to check the incoming data and call the other command when the input is only 1 line
The following code starts cat when it is ONLY fet in 1 line
sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' cat
How this works
Try reading a line, if failed go to step 5
If variable $LINE is NOT empty, goto step 6
Save line inside variable $LINE
Goto step 1
If $LINE is NOT empty, goto step 7
Exit the program with status code 1
Call our program and pass our $line to it using printf
Example usage:
Printing out only if grep found 1 match:
grep .... | sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' cat
Example of the question poster:
should_create_one_line | sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' expects_one_line
Here I could not find the number of words in the text file . What would be the possible changes do I need to make?
What is the use of tty in this program?
echo "Enter File name:"
read filename
terminal=`tty`
exec < $filename
num_line=0
num_words=0
while read line
do
num_lines=`expr $num_lines + 1`
num_words=`expr $num_words + 1`
done
There is a simple way using arrays to read the number of words in a file:
#!/bin/bash
[ -n "$1" ] || {
printf printf "error: insufficient input. Usage: %s\n" "${0//\//}"
exit 1
}
fn="$1"
[ -r "$fn" ] || {
printf "error: file not found: '%s'\n" "$fn"
exit 1
}
declare -i cnt=0
while read -r line || [ -n "$line" ]; do # read line from file
tmp=( $line ) # create tmp array of words
cnt=$((cnt + ${#tmp[#]})) # add no. of words to count
done <"$fn"
printf "\n %s words in %s\n\n" "$cnt" "$fn" # show results
exit 0
input:
$ cat dat/wordfile.txt
Here I could not find the number of words in the text file. What
would be the possible changes do I need to make? What is the use
of tty in this program?
output:
$bash wcount.sh dat/wordfile.txt
33 words in dat/wordfile.txt
wc -w confirmation:
$ wc -w dat/wordfile.txt
33 dat/wordfile.txt
tty?
The use of terminal=tty assigns the terminal device for the current interactive shell to the terminal variable. (It is a way to determine which tty device you are connected to e.g. /dev/pts/4)
tty command prints the name of the terminal connected to the standard output. In the context of your program, it does nothing significant really, you might as well remove that line and run.
Regarding the number of words calculation, you would need to parse each line and find it using space as the delimiter. Currently the program just finds the number of lines $num_lines and uses the same calculation for $num_words.
I'm trying to parse a csv file I made with Google Spreadsheet. It's very simple for testing purposes, and is basically:
1,2
3,4
5,6
The problem is that the csv doesn't end in a newline character so when I cat the file in BASH, I get
MacBook-Pro:Desktop kkSlider$ cat test.csv
1,2
3,4
5,6MacBook-Pro:Desktop kkSlider$
I just want to read line by line in a BASH script using a while loop that every guide suggests, and my script looks like this:
while IFS=',' read -r last first
do
echo "$last $first"
done < test.csv
The output is:
MacBook-Pro:Desktop kkSlider$ ./test.sh
1 2
3 4
Any ideas on how I could have it read that last line and echo it?
Thanks in advance.
You can force the input to your loop to end with a newline thus:
#!/bin/bash
(cat test.csv ; echo) | while IFS=',' read -r last first
do
echo "$last $first"
done
Unfortunately, this may result in an empty line at the end of your output if the input already has a newline at the end. You can fix that with a little addition:
!/bin/bash
(cat test.csv ; echo) | while IFS=',' read -r last first
do
if [[ $last != "" ]] ; then
echo "$last $first"
fi
done
Another method relies on the fact that the values are being placed into the variables by the read but they're just not being output because of the while statement:
#!/bin/bash
while IFS=',' read -r last first
do
echo "$last $first"
done <test.csv
if [[ $last != "" ]] ; then
echo "$last $first"
fi
That one works without creating another subshell to modify the input to the while statement.
Of course, I'm assuming here that you want to do more inside the loop that just output the values with a space rather than a comma. If that's all you wanted to do, there are other tools better suited than a bash read loop, such as:
tr "," " " <test.csv
cat file |sed -e '${/^$/!s/$/\n/;}'| while IFS=',' read -r last first; do echo "$last $first"; done
If the last (unterminated) line needs to be processed differently from the rest, #paxdiablo's version with the extra if statement is the way to go; but if it's going to be handled like all the others, it's cleaner to process it in the main loop.
You can roll the "if there was an unterminated last line" into the main loop condition like this:
while IFS=',' read -r last first || [ -n "$last" ]
do
echo "$last $first"
done < test.csv
The following program reads a file and it intends to store the all values (each line) into a variable but doesn't store the last line. Why?
file.txt :
1
2
.
.
.
n
Code :
FileName=file.txt
if test -f $FileName # Check if the file exists
then
while read -r line
do
fileNamesListStr="$fileNamesListStr $line"
done < $FileName
fi
echo "$fileNamesListStr" // 1 2 3 ..... n-1 (but it should print up to n.)
Instead of reading line-by-line, why not read the whole file at once?
[ -f $FileName ] && fileNameListStr=$( tr '\n' ' ' < $FileName )
One probable cause is that there misses a newline after the last line n.
Use the following command to check it:
tail -1 file.txt
And the following fixes:
echo >> file.txt
If you really need to keep the last line without newline, I reorganized the while loop here.
#!/bin/bash
FileName=0
if test -f $FileName ; then
while [ 1 ] ; do
read -r line
if [ -z $line ] ; then
break
fi
fileNamesListStr="$fileNamesListStr $line"
done < $FileName
fi
echo "$fileNamesListStr"
The issue is that when the file does not end in a newline, read returns non-zero and the loop does not proceed. The read command will still read the data, but it will not process the loop. This means that you need to do further processing outside of the loop. You also probably want an array instead of a space separated string.
FileName=file.txt
if test -f $FileName # Check if the file exists
then
while read -r line
do
fileNamesListArr+=("$line")
done < $FileName
[[ -n $line ]] && fileNamesListArr+=("$line")
fi
echo "${fileNameListArr[#]}"
See the "My text files are broken! They lack their final newlines!" section of this article:
http://mywiki.wooledge.org/BashFAQ/001
As a workaround, before reading from the text file a newline can be appended to the file.
echo "\n" >> $file_path
This will ensure that all the lines that was previously in the file will be read. Now the file can be read line by line.