Supposing we have a file list.txt:
line 1
line 2
if I use this:
for line in $(cat list.txt)
do
echo $line"_"
done
Even if I do:
OLD_IFS=$IFS
$IFS='$'
I get:
line1
line2_
and not:
line1_
line2_
How can I solve the problem?
Ignoring your incorrect assignment $IFS='$', $ does not mean newline, linefeed or end-of-line. It means literal dollar sign.
To assign a line feed, use
IFS=$'\n'
However, do not attempt to use this to iterate over lines. Instead, use a while read loop, which will not expand globs or collapse empty lines:
while IFS= read -r line
do
echo "${line}_"
done < file
or with similar benefits, read the lines into an array with mapfile:
mapfile -t lines < file
for line in "${lines[#]}"
do
echo "${line}_"
done
Related
I got a problem with storing some lines in a bash shell variable.
There is a file that contains a string like this.
$ vim a.txt
------we-are-in-vim------
first line
second line
-------end-of-file-------
This file has two empty lines that placed on the end of it.
When I cat this file, I can see that a blank is printed!
$ cat a.txt
first line
second line
$
Well. now, I can easily imagine that I can put this in a bash variable.
Let's try!
$ VAR=`cat a.txt`
$ echo "${VAR}"
first line
second line
$
Ok. I did not rap the cat command's output with double quotation! ;)
$ VAR="`cat a.txt`"
$ echo "${VAR}"
first line
second line
$
Ok. let's try with printf built-in variable assignment feature!
$ printf -v VAR "`cat a.txt`"
$ echo "${VAR}"
first line
second line
$
....Ok let's try with the mapfile command!
$ mapfile < a.txt VAR
$ printf '%s' "${VAR[#]}"
first line
second line
$
The mapfile command worked, but this is exactly the same with cat!
$ VAR2=`printf '%s' "${VAR[#]}"`
$ echo "${VAR2}"
first line
second line
$
I have already tried 'changing IFS to nothing' in bash,
But the result is exactly the same!
How can I assign the string that has two empty lines on the end of it, to a bash variable?
The usual wisdom is to append a character and remove it after:
$ a="`cat a.txt; echo x`"; echo "${a%x}"
first line
second line
$
That's a work around for the POSIX (most shells) specified variable expansion.
POSIX require that trailing newlines should be removed.
removing sequences of one or more characters at the end of the substitution
The alternative, if reading from a file (instead of executing some command), is to do the reading directly with the shell. If the shell's read accepts the option -d (since bash 2.04):
$ IFS='' read -d '' VAR < a.txt
$ echo "$VAR"
Or read with the command readarray (aka mapfile, since bash 4.0 alpha) as you already found.
It is because that is how command-substitution in bash works!
See this man bash except under COMMAND SUBSTITUTION
Command substitution allows the output of a command to replace the command name. There are two forms:
$(command)
`command`
[..] Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command with any trailing newlines deleted. [..]
I have code that requires a response within a for loop.
Prior to the loop I set IFS="\n"
Within the loop echo -n is ignored (except for the last line).
Note: this is just an example of the behavior of echo -n
Example:
IFS='\n'
for line in `cat file`
do
echo -n $line
done
This outputs:
this is a test
this is a test
this is a test$
with the user prompt occuring only at the end of the last line.
Why is this occuring and is there a fix?
Neither IFS="\n" nor IFS='\n' set $IFS to a newline; instead they set it to literal \ followed by literal n.
You'd have to use an ANSI C-quoted string in order to assign an actual newline: IFS=$'\n'; alternatively, you could use a normal string literal that contains an actual newline (spans 2 lines).
Assigning literal \n had the effect that the output from cat file was not split into lines, because an actual newline was not present in $IFS; potentially - though not with your sample file content - the output could have been split into fields by embedded \ and n characters.
Without either, the entire file contents were passed at once, resulting in a single iteration of your for loop.
That said, your approach to looping over lines from a file is ill-advised; try something like the following instead:
while IFS= read -r line; do
echo -n "$line"
done < file
Never use for loops when parsing files in bash. Use while loops instead. Here is a really good tutorial on that.
http://mywiki.wooledge.org/BashFAQ/001
I'm learning bash and I saw this construction:
cat file | while IFS= read -r line;
do
...
done
Can anyone explain what IFS= does? I know it's input field separator, but why is it being set to nothing?
IFS does many things but you are asking about that particular loop.
The effect in that loop is to preserve leading and trailing white space in line. To illustrate, first observe with IFS set to nothing:
$ echo " this is a test " | while IFS= read -r line; do echo "=$line=" ; done
= this is a test =
The line variable contains all the white space it received on its stdin. Now, consider the same statement with the default IFS:
$ echo " this is a test " | while read -r line; do echo "=$line=" ; done
=this is a test=
In this version, the white space internal to the line is still preserved. But, the leading and trailing white space have been removed.
What does -r do in read -r?
The -r option prevents read from treating backslash as a special character.
To illustrate, we use two echo commands that supply two lines to the while loop. Observe what happens with -r:
$ { echo 'this \\ line is \' ; echo 'continued'; } | while IFS= read -r line; do echo "=$line=" ; done
=this \\ line is \=
=continued=
Now, observe what happens without -r:
$ { echo 'this \\ line is \' ; echo 'continued'; } | while IFS= read line; do echo "=$line=" ; done
=this \ line is continued=
Without -r, two changes happened. First, the double-backslash was converted to a single backslash. Second, the backslash on the end of the first line was interpreted as a line-continuation character and the two lines were merged into one.
In sum, if you want backslashes in the input to have special meaning, don't use -r. If you want backslashes in the input to be taken as plain characters, then use -r.
Multiple lines of input
Since read takes input one line at a time, IFS behaves affects each line of multiple line input in the same way that it affects single line input. -r behaves similarly with the exception that, without -r, multiple lines can be combined into one line using the trailing backslash as shown above.
The behavior with multiple line input, however, can be changed drastically using read's -d flag. -d changes the delimiter character that read uses to mark the end of an input line. For example, we can terminate lines with a tab character:
$ echo $'line one \n line\t two \n line three\t ends here'
line one
line two
line three ends here
$ echo $'line one \n line\t two \n line three\t ends here' | while IFS= read -r -d$'\t' line; do echo "=$line=" ; done
=line one
line=
= two
line three=
Here, the $'...' construct was used to enter special characters like newline, \n and tab, \t. Observe that with -d$'\t', read divides its input into "lines" based on tab characters. Anything after the final tab is ignored.
How to handle the most difficult file names
The most important use of the features described above is to process difficult file names. Since the one character that cannot appear in path/filenames is the null character, the null character can be used to separate a list of file names. As an example:
while IFS= read -r -d $'\0' file
do
# do something to each file
done < <(find ~/music -type f -print0)
I want to read a file line by line in Unix shell scripting. Line can contain leading and trailing spaces and i want to read those spaces also in the line.
I tried with "while read line" but read command is removing space characters from line :(
Example if line in file are:-
abcd efghijk
abcdefg hijk
line should be read as:-
1) "abcd efghijk"
2) " abcdefg hijk"
What I tried is this (which not worked):-
while read line
do
echo $line
done < file.txt
I want line including space and tab characters in it.
Please suggest a way.
Try this,
IFS=''
while read line
do
echo $line
done < file.txt
EDIT:
From man bash
IFS - The Internal Field Separator that is used for word
splitting after expansion and to split lines into words
with the read builtin command. The default value is
``<space><tab><newline>''
You want to read raw lines to avoid problems with backslashes in the input (use -r):
while read -r line; do
printf "<%s>\n" "$line"
done < file.txt
This will keep whitespace within the line, but removes leading and trailing whitespace. To keep those as well, set the IFS empty, as in
while IFS= read -r line; do
printf "%s\n" "$line"
done < file.txt
This now is an equivalent of cat < file.txt as long as file.txt ends with a newline.
Note that you must double quote "$line" in order to keep word splitting from splitting the line into separate words--thus losing multiple whitespace sequences.
I want to read a file line by line in Unix shell scripting. Line can contain leading and trailing spaces and i want to read those spaces also in the line.
I tried with "while read line" but read command is removing space characters from line :(
Example if line in file are:-
abcd efghijk
abcdefg hijk
line should be read as:-
1) "abcd efghijk"
2) " abcdefg hijk"
What I tried is this (which not worked):-
while read line
do
echo $line
done < file.txt
I want line including space and tab characters in it.
Please suggest a way.
Try this,
IFS=''
while read line
do
echo $line
done < file.txt
EDIT:
From man bash
IFS - The Internal Field Separator that is used for word
splitting after expansion and to split lines into words
with the read builtin command. The default value is
``<space><tab><newline>''
You want to read raw lines to avoid problems with backslashes in the input (use -r):
while read -r line; do
printf "<%s>\n" "$line"
done < file.txt
This will keep whitespace within the line, but removes leading and trailing whitespace. To keep those as well, set the IFS empty, as in
while IFS= read -r line; do
printf "%s\n" "$line"
done < file.txt
This now is an equivalent of cat < file.txt as long as file.txt ends with a newline.
Note that you must double quote "$line" in order to keep word splitting from splitting the line into separate words--thus losing multiple whitespace sequences.