File content into unix variable with newlines - bash

Question

I have a text file test.txt with the following content:
text1
text2
And I want to assign the content of the file to a UNIX variable, but when I do this:
testvar=$(cat test.txt)
echo $testvar
the result is:
text1 text2
instead of
text1
text2
Can someone suggest me a solution for this?

Answer 1

The assignment does not remove the newline characters, it's actually the echo doing this. You need simply put quotes around the string to maintain those newlines:
echo "$testvar"
This will give the result you want. See the following transcript for a demo:
pax> cat num1.txt ; x=$(cat num1.txt)
line 1
line 2
pax> echo $x ; echo '===' ; echo "$x"
line 1 line 2
===
line 1
line 2
The reason why newlines are replaced with spaces is not entirely to do with the echo command, rather it's a combination of things.
When given a command line, bash splits it into words according to the documentation for the IFS variable:
IFS: The Internal Field Separator that is used for word splitting after expansion ... the default value is <space><tab><newline>.
That specifies that, by default, any of those three characters can be used to split your command into individual words. After that, the word separators are gone, all you have left is a list of words.
Combine that with the echo documentation (a bash internal command), and you'll see why the spaces are output:
echo [-neE] [arg ...]: Output the args, separated by spaces, followed by a newline.
When you use echo "$x", it forces the entire x variable to be a single word according to bash, hence it's not split. You can see that with:
pax> function count {
...> echo $#
...> }
pax> count 1 2 3
3
pax> count a b c d
4
pax> count $x
4
pax> count "$x"
1
Here, the count function simply prints out the number of arguments given. The 1 2 3 and a b c d variants show it in action.
Then we try it with the two variations on the x variable. The one without quotes shows that there are four words, "test", "1", "test" and "2". Adding the quotes makes it one single word "test 1\ntest 2".

Answer 2

This is due to IFS (Internal Field Separator) variable which contains newline.
$ cat xx1
1
2
$ A=`cat xx1`
$ echo $A
1 2
$ echo "|$IFS|"
|
|
A workaround is to reset IFS to not contain the newline, temporarily:
$ IFSBAK=$IFS
$ IFS=" "
$ A=`cat xx1` # Can use $() as well
$ echo $A
1
2
$ IFS=$IFSBAK
To REVERT this horrible change for IFS:
IFS=$IFSBAK

Answer 3

Bash -ge 4 has the mapfile builtin to read lines from the standard input into an array variable.
help mapfile
mapfile < file.txt lines
printf "%s" "${lines[#]}"
mapfile -t < file.txt lines # strip trailing newlines
printf "%s\n" "${lines[#]}"
See also:
http://bash-hackers.org/wiki/doku.php/commands/builtin/mapfile

Answer 4

Your variable is set correctly by testvar=$(cat test.txt). To display this variable which consist new line characters, simply add double quotes, e.g.
echo "$testvar"
Here is the full example:
$ printf "test1\ntest2" > test.txt
$ testvar=$(<test.txt)
$ grep testvar <(set)
testvar=$'test1\ntest2'
$ echo "$testvar"
text1
text2
$ printf "%b" "$testvar"
text1
text2

Answer 5

Just if someone is interested in another option:
content=( $(cat test.txt) )
a=0
while [ $a -le ${#content[#]} ]
do
echo ${content[$a]}
a=$[a+1]
done

Answer 6

The envdir utility provides an easy way to do this. envdir uses files to represent environment variables, with file names mapping to env var names, and file contents mapping to env var values. If the file contents contain newlines, so will the env var.
See https://pypi.python.org/pypi/envdir