Take care of last newline while capturing commands output [duplicate] - bash

This question already has answers here:
Capturing multiple line output into a Bash variable
(7 answers)
Closed 6 years ago.
I know $(cmd) captures the output of cmd into a string. However, it doesn't take care of the newline characters at the end.
Here's a demonstration:
a=$(echo x)
b=$(echo -n x)
[ "$a" = "$b" ] && echo equal
a and b capture outputs differed by a "\n" but these variables have the same value.
So my goal is still to capture the output of a command, but this time I want to preserve the last newline character(s) if there are any.

Trailing newlines are removed
POSIX requires that the $(…) notation (or the equivalent using back-ticks) strips all trailing newlines from the end of the string that is captured.
§6.2.3 Command substitution
…, removing sequences of one or more <newline> characters at the end of the substitution.
There isn't a simple way around that, or to detect how many newlines were deleted.
Add a single newline to the end of the output
If you have Bash 4.x (4.3 tested) you can play with shell parameter expansion and the substring notation, and add a dummy line of output to the end of the original string (that's the echo n in this example):
$ x=$(echo pandemonium; blanklines 4; echo n)
$ echo "$x"
pandemonium
n
$ y=${x: 0: -1}
$ echo "$y"
pandemonium
$
When using Bash 3.2 (GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)) on macOS Sierra, the expansion for y generates an error:
$ y=${x: 0: -1}
-bash: -1: substring expression < 0
$
See also Capturing multiple line output to a Bash variable.
Reliable and POSIX-compliant
Or you can use a simpler, more portable (POSIX-compatible) substitution suggested by the answer — as applied to this answer:
$ y=${x%n}
$ echo "$y"
pandemonium
$
Given that this works in strict POSIX shells and both Bash 3.x and 4.x, there's no need for the substring variant.

Related

print newline in process substitution [duplicate]

This question already has answers here:
Printf example in bash does not create a newline
(9 answers)
Closed 3 years ago.
This is in bash(5.0.3(1)-release) / ubuntu 19
When I ran :
printf "%s" $'\n'
It printed out a new line.
Now when I run:
result="$(printf "%s" $'\n')"
printf "<%s>" "$result"
I expect $result contains a newline, but it's empty.
Can someone explain ?
From posix shell manual, from chapter about process substitution, emphasis mine:
The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more <newline>s at the end of the substitution.
It's impossible to store trailing newlines with process substitution. You can use for example printf -v bash extension:
printf -v result "%s" $'\n'
For most portability, I go with encoding the string in hex using od or xxd and store the string in hex. Or just store the resulting string in a file.

How to get value from file in sh? [duplicate]

This question already has answers here:
How to read a file into a variable in shell?
(9 answers)
Difference between sh and Bash
(11 answers)
Closed 4 years ago.
Setup:
File a contains:
22
File b contains:
12
I have shell script 1.sh:
#!/bin/sh
a=$(< a)
b=$(< b)
echo $(($a*$b)) > c
The script should get values from file a and b, multiply them *, and save to file c.
However after setting permission $ chmod a+rx 1.sh and running it $ ./1.sh it returns an error:
./1.sh: 5: ./1.sh: arithmetic expression: expecting primary: "*"
This error occurs because the variables $a and $b doesn't get value form files a and b.
If I echo $a and echo $b it returns nothing;
If I define a=22 and b=12 values in the script it works;
I also tried other ways of getting contents of files like a=$(< 'a'), a=$(< "a"), a=$(< "~/a"), and even a=$(< cat a). None of those worked.
Plot Twist:
However, if I change shebang line to #!/bin/bash so that Bash shell is used - it works.
Question:
How to properly get data from file in sh?
Ignore everything from file a and b but numbers:
#!/bin/sh
a=$(tr -cd 0-9 < a)
b=$(tr -cd 0-9 < b)
echo $(($a*$b))
See: man tr
If you're looking for "true" Bourne-Shell compatibility, as opposed to Bash's emulation, then you have to go old school:
#!/bin/sh
a=`cat a`
b=`cat b`
expr $a \* $b > c
I tried your original example under #!/bin/sh on both macOS and Linux (FC26), and it behaved properly, assuming a and b had UNIX line-endings. If that can't be guaranteed, and you need to run under #!/bin/sh (as emulated by bash), then something like this will work:
#!/bin/sh
a=$(<a)
b=$(<b)
echo $(( ${a%%[^0-9]*} * ${b%%[^0-9]*} )) > c
There are many ways. One obvious way is to pipe in a sub-process by Command Substitution:
A=$(cat fileA.txt) # 22
B=$(cat fileB.txt) # 12
echo $((A*B))
# <do it in your head!>
If there are any other problems with multiple lines, you need to look into how to use the Bash variable $IFS (Internal File Separator). Usually IFS is defined by: IFS=$' \t\n', so if you need to be able to reliably read lines endings from both Windows and Linux EOL's you may need to modify it.
ADDENDUM:
Process Substitution
Bash, Zsh, and AT&T ksh{88,93} (but not pdksh/mksh) support process
substitution. Process substitution isn't specified by POSIX. You may
use NamedPipes to accomplish the same things. Coprocesses can also do
everything process substitutions can, and are slightly more portable
(though the syntax for using them is not).
This also means that most Android OS does not allow process substitution, since their shells are most often based on mksh.
From man bash:
Process Substitution
Process substitution allows a process's input or output to be referred to using a filename. It takes the form of <(list) or >(list). The
process list is run asynchronously, and its input or output appears as a filename. This filename is passed as an argument to the current
command as the result of the expansion. If the >(list) form is used, writing to the file will provide input for list. If the <(list) form
is used, the file passed as an argument should be read to obtain the output of list. Process substitution is supported on systems that sup-
port named pipes (FIFOs) or the /dev/fd method of naming open files.
When available, process substitution is performed simultaneously with parameter and variable expansion, command substitution, and arithmetic
expansion.

perl shasum vs bash shasum

The following bash and Perl scripts mysteriously give different results. Why?
#!/bin/bash
hash=`echo -n 'abcd' | /usr/bin/shasum -a 256`;
echo $hash;
#!/usr/bin/perl
$hash = `echo -n 'abcd' | /usr/bin/shasum -a 256`;
print "$hash";
The bash script:
$ ./tst.sh
88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589 -
The Perl script:
$ ./tst.pl
61799467ee1ab1f607764ab36c061f09cfac2f9c554e13f4c7442e66cbab9403 -
the heck?
Summary: In your Perl script, -n is being treated as an argument to include in the output of echo, not a flag to suppress the newline. ( Try
$hash = `echo -n 'abcd'`;
to confirm). Use printf instead.
Perl uses /bin/sh to execute code in back tics. Even if /bin/sh is a link to bash, it will behave differently when invoked via that like. In POSIX mode,
echo -n 'abcd'
will output
-n abcd
that is, the -n option is not recognized as a flag to suppress a newline, but is treated as a regular argument to print. Replace echo -n with printf in each script, and you should get the same SHA hash from each script.
(UPDATE: bash 3.2, when invoked as sh, displays this behavior. Newer versions of bash seem to continue treating -n as a flag when invoked as sh.)
Even better, don't shell out to do things you can do in Perl.
use Digest::SHA;
$hash = Digest::SHA::sha256('abcd');
For the curious, here's what the POSIX spec has to say about echo. I'm not sure what to make of XSI conformance; bash echo requires the -e option to treat the escape characters specially, but nearly every shell—except old versions of bash, and then only under special circumstances—treats -n as a flag, not a string. Oh well.
The following operands shall be supported:
string
A string to be written to standard output. If the first operand is -n, or
if any of the operands contain a <backslash> character, the results are
implementation-defined.
On XSI-conformant systems, if the first operand is -n, it shall be treated
as a string, not an option. The following character sequences shall be
recognized on XSI-conformant systems within any of the arguments:
\a
Write an <alert>.
\b
Write a <backspace>.
\c
Suppress the <newline> that otherwise follows the final argument in the output. All characters following the '\c' in the arguments shall be ignored.
\f
Write a <form-feed>.
\n
Write a <newline>.
\r
Write a <carriage-return>.
\t
Write a <tab>.
\v
Write a <vertical-tab>.
\\
Write a <backslash> character.
\0num
Write an 8-bit value that is the zero, one, two, or three-digit octal number num.
If you do:
printf "%s" 'abcd' | /usr/bin/shasum -a 256
you get the 88d...589 hash. If you do:
printf "%s\n" '-n abcd' | /usr/bin/shasum -a 256
you get the 617...403 hash.
Therefore, I deduce that Perl is somehow running a different echo command, perhaps /bin/echo or /usr/bin/echo instead of the bash built-in echo, or maybe the built-in echo to /bin/sh (which might perhaps be dash rather than bash), and this other echo does not recognize the -n option as an option and outputs different data.
I'm not sure which other echo it is finding; on my machine which is running an Ubuntu 14.04 LTE derivative, bash, dash, sh (linked to bash), ksh, csh and tcsh all treat echo -n abcd the same way. But somewhere along the line, I think that there is something along these lines happening; the hash checksums being identical strongly point to it. (Maybe you have a 3.2 bash linked to sh; see the notes in the comments.)

Newlines at the end "$( .... )" get removed in shell scripts. Why?

Can someone explain why the output of these two commands is different?
$ echo "NewLine1\nNewLine2\n"
NewLine1
NewLine2
<-- Note 2nd newline here
$ echo "$(echo "NewLine1\nNewLine2\n")"
NewLine1
NewLine2
$ <-- No second newline
Is there any good way that I can keep the new lines at the end of the output in "$( .... )" ? I've thought about just adding a dummy letter and removing it, but I'd quite like to understand why those new lines are going away.
Because that's what POSIX specifies and has always been like that in Bourne shells:
2.6.3 Command Substitution
Command substitution allows the output of a command to be substituted
in place of the command name itself. Command substitution shall occur
when the command is enclosed as follows:
$(command)
or (backquoted version):
`command`
The shell shall expand the command substitution by executing command
in a subshell environment (see Shell Execution Environment) and
replacing the command substitution (the text of command plus the
enclosing "$()" or backquotes) with the standard output of the
command, removing sequences of one or more <newline> characters at the
end of the substitution. Embedded <newline> characters before the end
of the output shall not be removed; however, they may be treated as
field delimiters and eliminated during field splitting, depending on
the value of IFS and quoting that is in effect. If the output contains
any null bytes, the behavior is unspecified.
One way to keep the final newline(s) would be
VAR="$(command; echo x)" # Append x to keep newline(s).
VAR=${VAR%x} # Chop x.
Vis.:
$ x="$(whoami; echo x)" ; printf '<%s>\n' "$x" "${x%x}"
<user
x>
<user
>
But why remove trailing newlines? Because more often than not you want it that way. I'm also programming in perl and I can't count the number of times where I read a line or variable and then need to chop the newline:
while (defined ($string = <>)) {
chop $string;
frobnitz($string);
}
command substitution removes every trailing newline.
It makes sense to remove one. For instance:
basename foo/bar
outputs bar\n. In:
var=$(basename foo/bar)
you want $var to contain bar, not bar\n.
However in
var=$(basename $'foo/bar\n')
You would like $var to contain bar\n (after all, newline is as valid a character as any in a file name on Unix). But all shells remove every trailing newline character. That misfeature was in the original Bourne shell and even rc which has fixed most of Bourne's flaws has not fixed that one. (though rc has the ``(){cmd} syntax to not strip any newline character).
In POSIX shells, to work around the issue, you can do:
var=$(basename -- "$file"; echo .)
var=${var%??}
Though you're then losing the exit status of basename. Which you can fix with:
var=$(basename -- "$file" && echo .) && var=${var%??}
${var%??} is to remove the last two characters. The first one is the . that we added above, the second is the one newline character added by basename, we're not removing any more as command substitution would do as the other newline characters, if any, would be part of the filename we want to get the base of, so we do want them.
In the Bourne shell which doesn't have the ${var%x} operator, you had to go a long and convoluted way to work around it.
If the newlines were not removed, then constructs like:
x="$(pwd)/filename"
would not work usefully, but the people who wrote Unix preferred useful behaviour.
Once, briefly, a very long time ago (like 1983, maybe 1984), I suffered from a shell update on a particular variant of Unix that didn't remove the trailing newline. It broke scripts all over the place. It was fixed very quickly.

Semicolons superfluous at the end of a line in shell scripts? [duplicate]

This question already has answers here:
Bash semicolon being equal to newline is not exactly true?
(4 answers)
Closed 2 years ago.
I have a shell script which contains the following:
case $1 in
0 )
echo $1 = 0;
OUTPUT=3;;
1 )
echo $1 = 1;
OUTPUT=4;;
2 )
echo $1 = 2;
OUTPUT=4;;
esac
HID=$2;
BUNCH=16;
LR=.008;
Are semicolons completely superfluous in the snippet above? And is there any reason for some people using double semicolons?
It appears semicolons are only a separator, something you would use instead of a new line.
Single semicolons at the end of a line are superfluous, since the newline is also a command separator. case specifically needs double semicolons at the end of the last command in each pattern block; see help case for details.
According to man bash:
metacharacter
A character that, when unquoted, separates words. One of the following:
| & ; ( ) < > space tab
control operator
A token that performs a control function. It is one of the following symbols:
|| & && ; ;; ( ) | |& <newline>
So, the ; can be metacharacter or control operator, while the ;; is always a control operator (in case command).
In your particular code, all ; at the end of line are not needed. The ;; is needed however.
In the special case of find, ; is used to terminate commands invoked by -exec. See the answer of #kenorb to this question.
#Opensourcebook-Amit
newlines equivalent to single semicolon ; on terminal or in shell script.
See the below examples:
On terminal:
[root#server test]# ls;pwd;
On shell script:
[root#server test]# cat test4.sh
echo "Current UserName:"
whoami
echo -e "\nCurrent Date:";date;
[root#server test]#
But I am not agree with the comment that & is equivalent to newline or single semicolon
& is run commands in background also a command separator but not worked as semicolon or newline.

Resources