I have file a.txt with following content
aaa
bbb
When I execute following script:
while read line
do
echo $line
done < a.txt > b.txt
generated b.txt contains following
aaa
bbb
It is seen that the leading spaces of lines have got removed. How can I preserve leading spaces?
This is covered in the Bash FAQ entry on reading data line-by-line.
The read command modifies each line read; by default it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS). If that is not desired, the IFS variable has to be cleared:
# Exact lines, no trimming
while IFS= read -r line; do
printf '%s\n' "$line"
done < "$file"
As Charles Duffy correctly points out (and I'd missed by focusing on the IFS issue); if you want to see the spaces in your output you also need to quote the variable when you use it or the shell will, once again, drop the whitespace.
Notes about some of the other differences in that quoted snippet as compared to your original code.
The use of the -r argument to read is covered in a single sentence at the top of the previously linked page.
The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should almost always use the -r option with read.
As to using printf instead of echo there the behavior of echo is, somewhat unfortunately, not portably consistent across all environments and the differences can be awkward to deal with. printf on the other hand is consistent and can be used entirely robustly.
There are several problems here:
Unless IFS is cleared, read strips leading and trailing whitespace.
echo $line string-splits and glob-expands the contents of $line, breaking it up into individual words, and passing those words as individual arguments to the echo command. Thus, even with IFS cleared at read time, echo $line would still discard leading and trailing whitespace, and change runs of whitespace between words into a single space character each. Additionally, a line containing only the character * would be expanded to contain a list of filenames.
echo "$line" is a significant improvement, but still won't correctly handle values such as -n, which it treats as an echo argument itself. printf '%s\n' "$line" would fix this fully.
read without -r treats backslashes as continuation characters rather than literal content, such that they won't be included in the values produced unless doubled-up to escape themselves.
Thus:
while IFS= read -r line; do
printf '%s\n' "$line"
done
Related
I have code that requires a response within a for loop.
Prior to the loop I set IFS="\n"
Within the loop echo -n is ignored (except for the last line).
Note: this is just an example of the behavior of echo -n
Example:
IFS='\n'
for line in `cat file`
do
echo -n $line
done
This outputs:
this is a test
this is a test
this is a test$
with the user prompt occuring only at the end of the last line.
Why is this occuring and is there a fix?
Neither IFS="\n" nor IFS='\n' set $IFS to a newline; instead they set it to literal \ followed by literal n.
You'd have to use an ANSI C-quoted string in order to assign an actual newline: IFS=$'\n'; alternatively, you could use a normal string literal that contains an actual newline (spans 2 lines).
Assigning literal \n had the effect that the output from cat file was not split into lines, because an actual newline was not present in $IFS; potentially - though not with your sample file content - the output could have been split into fields by embedded \ and n characters.
Without either, the entire file contents were passed at once, resulting in a single iteration of your for loop.
That said, your approach to looping over lines from a file is ill-advised; try something like the following instead:
while IFS= read -r line; do
echo -n "$line"
done < file
Never use for loops when parsing files in bash. Use while loops instead. Here is a really good tutorial on that.
http://mywiki.wooledge.org/BashFAQ/001
I have file a.txt with following content
aaa
bbb
When I execute following script:
while read line
do
echo $line
done < a.txt > b.txt
generated b.txt contains following
aaa
bbb
It is seen that the leading spaces of lines have got removed. How can I preserve leading spaces?
This is covered in the Bash FAQ entry on reading data line-by-line.
The read command modifies each line read; by default it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS). If that is not desired, the IFS variable has to be cleared:
# Exact lines, no trimming
while IFS= read -r line; do
printf '%s\n' "$line"
done < "$file"
As Charles Duffy correctly points out (and I'd missed by focusing on the IFS issue); if you want to see the spaces in your output you also need to quote the variable when you use it or the shell will, once again, drop the whitespace.
Notes about some of the other differences in that quoted snippet as compared to your original code.
The use of the -r argument to read is covered in a single sentence at the top of the previously linked page.
The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should almost always use the -r option with read.
As to using printf instead of echo there the behavior of echo is, somewhat unfortunately, not portably consistent across all environments and the differences can be awkward to deal with. printf on the other hand is consistent and can be used entirely robustly.
There are several problems here:
Unless IFS is cleared, read strips leading and trailing whitespace.
echo $line string-splits and glob-expands the contents of $line, breaking it up into individual words, and passing those words as individual arguments to the echo command. Thus, even with IFS cleared at read time, echo $line would still discard leading and trailing whitespace, and change runs of whitespace between words into a single space character each. Additionally, a line containing only the character * would be expanded to contain a list of filenames.
echo "$line" is a significant improvement, but still won't correctly handle values such as -n, which it treats as an echo argument itself. printf '%s\n' "$line" would fix this fully.
read without -r treats backslashes as continuation characters rather than literal content, such that they won't be included in the values produced unless doubled-up to escape themselves.
Thus:
while IFS= read -r line; do
printf '%s\n' "$line"
done
I want to read a file line by line in Unix shell scripting. Line can contain leading and trailing spaces and i want to read those spaces also in the line.
I tried with "while read line" but read command is removing space characters from line :(
Example if line in file are:-
abcd efghijk
abcdefg hijk
line should be read as:-
1) "abcd efghijk"
2) " abcdefg hijk"
What I tried is this (which not worked):-
while read line
do
echo $line
done < file.txt
I want line including space and tab characters in it.
Please suggest a way.
Try this,
IFS=''
while read line
do
echo $line
done < file.txt
EDIT:
From man bash
IFS - The Internal Field Separator that is used for word
splitting after expansion and to split lines into words
with the read builtin command. The default value is
``<space><tab><newline>''
You want to read raw lines to avoid problems with backslashes in the input (use -r):
while read -r line; do
printf "<%s>\n" "$line"
done < file.txt
This will keep whitespace within the line, but removes leading and trailing whitespace. To keep those as well, set the IFS empty, as in
while IFS= read -r line; do
printf "%s\n" "$line"
done < file.txt
This now is an equivalent of cat < file.txt as long as file.txt ends with a newline.
Note that you must double quote "$line" in order to keep word splitting from splitting the line into separate words--thus losing multiple whitespace sequences.
I want to read a file line by line in Unix shell scripting. Line can contain leading and trailing spaces and i want to read those spaces also in the line.
I tried with "while read line" but read command is removing space characters from line :(
Example if line in file are:-
abcd efghijk
abcdefg hijk
line should be read as:-
1) "abcd efghijk"
2) " abcdefg hijk"
What I tried is this (which not worked):-
while read line
do
echo $line
done < file.txt
I want line including space and tab characters in it.
Please suggest a way.
Try this,
IFS=''
while read line
do
echo $line
done < file.txt
EDIT:
From man bash
IFS - The Internal Field Separator that is used for word
splitting after expansion and to split lines into words
with the read builtin command. The default value is
``<space><tab><newline>''
You want to read raw lines to avoid problems with backslashes in the input (use -r):
while read -r line; do
printf "<%s>\n" "$line"
done < file.txt
This will keep whitespace within the line, but removes leading and trailing whitespace. To keep those as well, set the IFS empty, as in
while IFS= read -r line; do
printf "%s\n" "$line"
done < file.txt
This now is an equivalent of cat < file.txt as long as file.txt ends with a newline.
Note that you must double quote "$line" in order to keep word splitting from splitting the line into separate words--thus losing multiple whitespace sequences.
I'm doing the following, which basically works.
The script tries to insert some lines into a file to rewrite it.
But it is stripping all blank lines and also all line padding.
The main problem is that it does not process the last line of the file.
I'm not sure why.
while read line; do
<... process some things ...>
echo ${line}>> "${ACTION_PATH_IN}.work"
done < "${ACTION_PATH_IN}"
What can be done to fix this?
while IFS= read -r line; do
## some work
printf '%s\n' "$line" >> output
done < <(printf '%s\n' "$(cat input)")
An empty IFS tells read not to strip leading and trailing whitespace.
read -r prevents backslash at EOL from creating a line continuation.
Double-quote your parameter substitution ("$line") to prevent the shell from doing word splitting and globbing on its value.
Use printf '%s\n' instead of echo because it is reliable when processing values like like -e, -n, etc.
< <(printf '%s\n' "$(cat input)") is an ugly way of LF terminating the contents of input. Other constructions are possible, depending on your requirements (pipe instead of redirect from process substitution if it is okay that your whole while runs in a subshell).
It might be better if you just ensured that it was LF-terminated before processing it.
Best yet, use a tool such as awk instead of the shell's while loop. First, awk is meant for parsing/manipulating files so for a huge file to process, awk has the advantage. Secondly, you won't have to care whether you have the last newline or not (for your case).
Hence the equivalent of your while read loop:
awk '{
# process lines
# print line > "newfile.txt"
}' file
One possible reason for not reading the last line is that the file does not end with a newline. On the whole, I'd expect it to work even so, but that could be why.
On MacOS X (10.7.1), I got this output, which is the behaviour you are seeing:
$ /bin/echo -n Hi
Hi$ /bin/echo -n Hi > x
$ while read line; do echo $line; done < x
$
The obvious fix is to ensure that the file ends with a newline.
First thing, use
echo "$line" >> ...
Note the quotes. If you don't put them, the shell itself will remove the padding.
As for the last line, it is strange. It may have to do with whether the last line of the file is terminated by a \n or not (it is a good practice to do so, and almost any editor will do that for you).