foo.sh:
#!/bin/bash
x="$(cat)"
echo "got >>>$x<<<"
In the shell
#> echo abc | foo.sh
got >>>abc<<<
Where has the trailing newline character gone?
Bash command substitution operator intentionally drops trailing newlines; this is documented in the manual:
Bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.
This behavior is not specific to Bash, it was done by Bourne-descended shells before it. The reason is convenience - since most programs' output ends with a newline, retaining them would require code that processes the output to take the newline into account.
Since newlines are only removed from the end, one can work around this behavior by adding a dummy character at the end and stripping it manually:
# preserve trailing newlines
$ a="$(echo foo; echo .)"; a=${a:0:-1}
$ echo ">>>$a<<<"
>>>foo
<<<
# also works for programs that don't output a newline
$ b="$(printf foo; echo .)"; b=${b:0:-1}
$ echo ">>>$b<<<"
>>>foo<<<
Refer to this answer for more sophisticated workarounds.
Related
This question already has answers here:
echo "#!" fails -- "event not found"
(5 answers)
Closed 7 years ago.
I am attempting to parse the output of a VNC server startup event and have run into a problem in parsing using sed in a command substitution. Specifically, the remote VNC server is started in a manner such as the following:
address1="user1#lxplus.cern.ch"
VNCServerResponse="$(ssh "${address1}" 'vncserver' 2>&1)"
The standard error output produced in this startup event is then to be parsed in order to extract the server and display information. At this point the content of the variable VNCServerResponse is something such as the following:
New 'lxplus0186.cern.ch:1 (user1)' desktop is lxplus0186.cern.ch:1
Starting applications specified in /afs/cern.ch/user/u/user1/.vnc/xstartup
Log file is /afs/cern.ch/user/u/user1/.vnc/lxplus0186.cern.ch:1.log
This output can be parsed in the following way in order to extract the server and display information:
echo "${VNCServerResponse}" | sed '/New.*desktop.*is/!d' \
| awk -F" desktop is " '{print $2}'
The result is something such as the following:
lxplus0186.cern.ch:1
What I want to do is use this parsing in a command substitution something like the following:
VNCServerAndDisplayNumber="$(echo "${VNCServerResponse}" \
| sed '/New.*desktop.*is/!d' | awk -F" desktop is " '{print $2}')"
On attempting to do this, I am presented with the following error:
bash: !d': event not found
I am not sure how to address this. It appears to be a problem in the way sed is being used in the command substitution. I would appreciate guidance.
Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.
How history expansion is enabled (or disabled)
There are two shell options which control history expansion:
set -o history: Required for the history to be recorded.
set -H (or set -o histexpand): Additionally required for history expansion to be enabled.
Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)
According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:
set -o history -o histexpand
The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]
History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:
set -o history
which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.
In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.
How history expansion is parsed
The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.
By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:
when it is followed by whitespace or =
if the shell option extglob is set, and it is followed by ( [Note 1]
if it appears in a single-quoted string
if it is preceded by a \ [Note 2 and see below]
if it is preceded by $ or ${ [Note 1]
if it is preceded by [ [Note 1]
(As of bash v4.3) if it is the last character in a double-quoted string.
The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:
$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s
However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:
# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST
So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:
result="$(sed '/foo/!d' file)"
you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "\!" inhibits history expansion, it doesn't remove the backslash:
$ echo "\!"
\!
In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:
# Undesired history expansion
printf "The answer is '%s'\n" "$(sed '/foo/!d' file)"
# Undesired word splitting
printf "The answer is '%s'\n" $(sed '/foo/!d' file)
In this case, the best solution is probably to put the sed argument in a variable
# Works
sed_prog='/foo/!d'
printf "The answer is '%s'\n" "$(sed "$sed_prog" file)"
(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)
Notes:
The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:
# No matching close parenthesis
$ echo "!("
bash: !: event not found
# The matching close parenthesis has nothing to do with the open
$ echo "!(" ")"
!( )
# An actual extended glob: files whose names don't start with a
$ echo "!(a*)"
b
As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:
$ echo \!
!
$ echo \\!
\!
$ echo \\\!
\!
\ also inhibits history expansion inside double quotes, but \! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:
$ echo "\!"
\!
$ echo "\\!"
\!
$ echo "\\\!"
\\!
I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.
The problem is that within double quotes, bash is trying to expand !d before passing it to the subshell. You can get around this problem by removing the double quotes but I would also propose a simplification to your script:
VNCServerAndDisplayNumber=$(echo "$VNCServerResponse" | awk '/desktop/ {print $NF}')
This simply prints the last field on the line containing the word "desktop".
On a newer bash, you can use a herestring rather than piping an echo:
VNCServerAndDisplayNumber=$(awk '/desktop/ {print $NF}' <<<"$VNCServerResponse")
Don't wrap the $(...) command substitution in double quotes. You are asking the shell to perform evaluation on the contents of the quotes and are hitting the history substitution expansion feature. Drop the quotes and you stop telling the shell to do that and you won't hit that problem.
And yes, dropping those quotes is safe on that assignment line even if the output may contain spaces or newlines or whatever. Assignments of that sort are not going to split on those the way command substitution or variable evaluation will on a normal shell execution line.
Alternatively, disable history expansion in your shell/script before you run that. (It should be off when running a script by default I believe anyway.)
This only happens when history expansion is enabled, which it normally isn't and definitely shouldn't be for scripts.
Rather than trying to work around it, figure out why history expansion is enabled and what to do so it isn't.
If you're executing your script with . foo or source foo, use ./foo instead.
If you're writing this as a function in .bashrc or similar, consider making it a separate script.
If your script (or BASH_ENV) explicitly does set -H, don't.
Quote it with '' or \ or disable history expansion with set +H or shopt -u -o histexpand. See History Expansion.
I would like to use the bash here doc syntax to build a long string. I would like the heredoc to ignore newlines/spaces/tabs even when I use newlines for code clarity.
I thought this would work:
#!/bin/bash
#http://unix.stackexchange.com/questions/20035/how-to-add-newlines-into-variables-in-bash-script
IFS= read -r -d '' NS_LOG<<-EOF
*=error|warn|prefix_node|prefix_func
:PointToPointNetDevice
:ClockTest
:ClockPerfect
:TcpTestSuite
:TcpRxBuffer
:TcpTxBuffer
:TcpHeader=*
:TcpL4Protocol
:TraceHelper:PointToPointHelper
EOF
echo $NS_LOG
export NS_LOG
but somewhere bash appends spaces between lines and instead of having the desired
*=error|warn|prefix_node|prefix_func:PointToPointNetDevice:ClockTest:ClockPerfect:Clock
I have when running $ ./launch_myscript.sh:
*=error|warn|prefix_node|prefix_func :PointToPointNetDevice :ClockTest :ClockPerfect :Clock etc...
My bash --version:
GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu)
I just saw in the recommended posts this one Bash: Why is echo adding extra space?. How can I prevent NS_LOG from being considered as several arguments? Ultimately the goal is to export that variable.
Your read command is very explicitly treating newlines as data: By clearing IFS and passing -d '', you tell read not to treat whitespace characters as special; since they're not special, they go into the output variable like everything else. However, you can take them out later:
IFS= read -r -d '' NS_LOG <<'EOF'
...content...
EOF
NS_LOG=${NS_LOG//[[:space:]]/} ## replace all whitespace with the empty string
printf '%s\n' "$NS_LOG" ## the quotes are important!
See this snippet run, and its output, at http://ideone.com/fWhzBB.
Notes:
<<'EOF' prevents expansions from occurring within the heredoc itself; with <<EOF, $foo, $(foo), etc. would be special.
<<- only trims leading tab characters, not any other form of whitespace; it's typically safer to do without.
echo $foo string-splits and glob-expands the contents of $foo, passing each word created by this process as a separate argument; echo then places spaces between each argument. echo "$foo" ensures that the entire expansion is treated as a single word. See BashPitfalls #14.
Using echo with nontrivial or unknown data is advised against in the relevant portion of the POSIX specification; printf is the preferred substitute. POSIX echo is explicitly allowed to behave in undefined ways when content contains backslash literals, and the BSD- and AT&T-derived forms of the command are explicitly incompatible, both with each other and with the common GNU implementation (providing an -e flag, which the POSIX spec requires to simply print -e on its output).
things aren't that difficult. here is another solution:
var="`tr -d '[:space:] <<EOF'
your
text
with
lot of spaces
EOF`"
result:
$ echo "$var"
yourtextwithlotofspaces
I know there is a duplicate for this question already at: How to trim whitespace from a Bash variable?.
I read all the answers there but I have a question about another solution in my mind and I want to know if this works.
This is the solution I think works.
a=$(printf "%s" $a)
Here is a demonstration.
$ a=" foo "
$ a=$(printf "%s" $a)
$ echo "$a"
foo
Is there any scenario in which this solution may fail?
If there is such a scenario in which this solution may fail, can we modify this solution to handle that scenario without compromising the simplicity of the solution too much?
If the variable a is set with something like "-e", "-n" in the begining, depending on how you process later your result, a user might crash your script:
-e option allows echo to interpret things backslashed.
Even in the case you only want to display the variable a, -n would screw your layout.
You could think about using regex to check if your variable starts with '-' and is followed by one of the available echo options (-n, -e, -E, --help, --version).
It fails when the input contains spaces between non-whitespace characters.
$ a=" foo bar "
$ a=$(printf "%s" $a)
$ echo "$a"
foobar
The expected output was the following instead.
foo bar
You could use Bash's builtin pattern substitution.
Note: Bash pattern substitution uses 'Pathname Expansion' (glob) pattern matching, not regular expressions. My solution requires enabling the optional shell behaviour extglob (shopt -s extglob).
$shopt -s extglob
$ a=" foo bar "
$ echo "Remove trailing spaces: '${a/%*([[:space:]])}'"
Remove trailing spaces: ' foo bar'
$ echo "Remove leading spaces: '${a/#*([[:space:]])}'"
Remove leading spaces: 'foo bar '
$ echo "Remove all spaces anywhere: '${a//[[:space:]]}'"
Remove all spaces anywhere: 'foobar'
For reference, refer to the 'Parameter Expansion' (Pattern substitution) and 'Pathname Expansion' subsections of the EXPANSION section of the Bash man page.
This question already has answers here:
echo "#!" fails -- "event not found"
(5 answers)
Closed 7 years ago.
I am attempting to parse the output of a VNC server startup event and have run into a problem in parsing using sed in a command substitution. Specifically, the remote VNC server is started in a manner such as the following:
address1="user1#lxplus.cern.ch"
VNCServerResponse="$(ssh "${address1}" 'vncserver' 2>&1)"
The standard error output produced in this startup event is then to be parsed in order to extract the server and display information. At this point the content of the variable VNCServerResponse is something such as the following:
New 'lxplus0186.cern.ch:1 (user1)' desktop is lxplus0186.cern.ch:1
Starting applications specified in /afs/cern.ch/user/u/user1/.vnc/xstartup
Log file is /afs/cern.ch/user/u/user1/.vnc/lxplus0186.cern.ch:1.log
This output can be parsed in the following way in order to extract the server and display information:
echo "${VNCServerResponse}" | sed '/New.*desktop.*is/!d' \
| awk -F" desktop is " '{print $2}'
The result is something such as the following:
lxplus0186.cern.ch:1
What I want to do is use this parsing in a command substitution something like the following:
VNCServerAndDisplayNumber="$(echo "${VNCServerResponse}" \
| sed '/New.*desktop.*is/!d' | awk -F" desktop is " '{print $2}')"
On attempting to do this, I am presented with the following error:
bash: !d': event not found
I am not sure how to address this. It appears to be a problem in the way sed is being used in the command substitution. I would appreciate guidance.
Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.
How history expansion is enabled (or disabled)
There are two shell options which control history expansion:
set -o history: Required for the history to be recorded.
set -H (or set -o histexpand): Additionally required for history expansion to be enabled.
Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)
According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:
set -o history -o histexpand
The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]
History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:
set -o history
which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.
In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.
How history expansion is parsed
The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.
By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:
when it is followed by whitespace or =
if the shell option extglob is set, and it is followed by ( [Note 1]
if it appears in a single-quoted string
if it is preceded by a \ [Note 2 and see below]
if it is preceded by $ or ${ [Note 1]
if it is preceded by [ [Note 1]
(As of bash v4.3) if it is the last character in a double-quoted string.
The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:
$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s
However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:
# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST
So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:
result="$(sed '/foo/!d' file)"
you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "\!" inhibits history expansion, it doesn't remove the backslash:
$ echo "\!"
\!
In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:
# Undesired history expansion
printf "The answer is '%s'\n" "$(sed '/foo/!d' file)"
# Undesired word splitting
printf "The answer is '%s'\n" $(sed '/foo/!d' file)
In this case, the best solution is probably to put the sed argument in a variable
# Works
sed_prog='/foo/!d'
printf "The answer is '%s'\n" "$(sed "$sed_prog" file)"
(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)
Notes:
The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:
# No matching close parenthesis
$ echo "!("
bash: !: event not found
# The matching close parenthesis has nothing to do with the open
$ echo "!(" ")"
!( )
# An actual extended glob: files whose names don't start with a
$ echo "!(a*)"
b
As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:
$ echo \!
!
$ echo \\!
\!
$ echo \\\!
\!
\ also inhibits history expansion inside double quotes, but \! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:
$ echo "\!"
\!
$ echo "\\!"
\!
$ echo "\\\!"
\\!
I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.
The problem is that within double quotes, bash is trying to expand !d before passing it to the subshell. You can get around this problem by removing the double quotes but I would also propose a simplification to your script:
VNCServerAndDisplayNumber=$(echo "$VNCServerResponse" | awk '/desktop/ {print $NF}')
This simply prints the last field on the line containing the word "desktop".
On a newer bash, you can use a herestring rather than piping an echo:
VNCServerAndDisplayNumber=$(awk '/desktop/ {print $NF}' <<<"$VNCServerResponse")
Don't wrap the $(...) command substitution in double quotes. You are asking the shell to perform evaluation on the contents of the quotes and are hitting the history substitution expansion feature. Drop the quotes and you stop telling the shell to do that and you won't hit that problem.
And yes, dropping those quotes is safe on that assignment line even if the output may contain spaces or newlines or whatever. Assignments of that sort are not going to split on those the way command substitution or variable evaluation will on a normal shell execution line.
Alternatively, disable history expansion in your shell/script before you run that. (It should be off when running a script by default I believe anyway.)
This only happens when history expansion is enabled, which it normally isn't and definitely shouldn't be for scripts.
Rather than trying to work around it, figure out why history expansion is enabled and what to do so it isn't.
If you're executing your script with . foo or source foo, use ./foo instead.
If you're writing this as a function in .bashrc or similar, consider making it a separate script.
If your script (or BASH_ENV) explicitly does set -H, don't.
Quote it with '' or \ or disable history expansion with set +H or shopt -u -o histexpand. See History Expansion.
Can someone explain why the output of these two commands is different?
$ echo "NewLine1\nNewLine2\n"
NewLine1
NewLine2
<-- Note 2nd newline here
$ echo "$(echo "NewLine1\nNewLine2\n")"
NewLine1
NewLine2
$ <-- No second newline
Is there any good way that I can keep the new lines at the end of the output in "$( .... )" ? I've thought about just adding a dummy letter and removing it, but I'd quite like to understand why those new lines are going away.
Because that's what POSIX specifies and has always been like that in Bourne shells:
2.6.3 Command Substitution
Command substitution allows the output of a command to be substituted
in place of the command name itself. Command substitution shall occur
when the command is enclosed as follows:
$(command)
or (backquoted version):
`command`
The shell shall expand the command substitution by executing command
in a subshell environment (see Shell Execution Environment) and
replacing the command substitution (the text of command plus the
enclosing "$()" or backquotes) with the standard output of the
command, removing sequences of one or more <newline> characters at the
end of the substitution. Embedded <newline> characters before the end
of the output shall not be removed; however, they may be treated as
field delimiters and eliminated during field splitting, depending on
the value of IFS and quoting that is in effect. If the output contains
any null bytes, the behavior is unspecified.
One way to keep the final newline(s) would be
VAR="$(command; echo x)" # Append x to keep newline(s).
VAR=${VAR%x} # Chop x.
Vis.:
$ x="$(whoami; echo x)" ; printf '<%s>\n' "$x" "${x%x}"
<user
x>
<user
>
But why remove trailing newlines? Because more often than not you want it that way. I'm also programming in perl and I can't count the number of times where I read a line or variable and then need to chop the newline:
while (defined ($string = <>)) {
chop $string;
frobnitz($string);
}
command substitution removes every trailing newline.
It makes sense to remove one. For instance:
basename foo/bar
outputs bar\n. In:
var=$(basename foo/bar)
you want $var to contain bar, not bar\n.
However in
var=$(basename $'foo/bar\n')
You would like $var to contain bar\n (after all, newline is as valid a character as any in a file name on Unix). But all shells remove every trailing newline character. That misfeature was in the original Bourne shell and even rc which has fixed most of Bourne's flaws has not fixed that one. (though rc has the ``(){cmd} syntax to not strip any newline character).
In POSIX shells, to work around the issue, you can do:
var=$(basename -- "$file"; echo .)
var=${var%??}
Though you're then losing the exit status of basename. Which you can fix with:
var=$(basename -- "$file" && echo .) && var=${var%??}
${var%??} is to remove the last two characters. The first one is the . that we added above, the second is the one newline character added by basename, we're not removing any more as command substitution would do as the other newline characters, if any, would be part of the filename we want to get the base of, so we do want them.
In the Bourne shell which doesn't have the ${var%x} operator, you had to go a long and convoluted way to work around it.
If the newlines were not removed, then constructs like:
x="$(pwd)/filename"
would not work usefully, but the people who wrote Unix preferred useful behaviour.
Once, briefly, a very long time ago (like 1983, maybe 1984), I suffered from a shell update on a particular variant of Unix that didn't remove the trailing newline. It broke scripts all over the place. It was fixed very quickly.