This question already has answers here:
Printf example in bash does not create a newline
(9 answers)
Closed 3 years ago.
This is in bash(5.0.3(1)-release) / ubuntu 19
When I ran :
printf "%s" $'\n'
It printed out a new line.
Now when I run:
result="$(printf "%s" $'\n')"
printf "<%s>" "$result"
I expect $result contains a newline, but it's empty.
Can someone explain ?
From posix shell manual, from chapter about process substitution, emphasis mine:
The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more <newline>s at the end of the substitution.
It's impossible to store trailing newlines with process substitution. You can use for example printf -v bash extension:
printf -v result "%s" $'\n'
For most portability, I go with encoding the string in hex using od or xxd and store the string in hex. Or just store the resulting string in a file.
Related
This question already has answers here:
How to avoid bash command substitution to remove the newline character?
(3 answers)
Closed 3 years ago.
I'm dealing with a while loop processing a Herestring input:
one_is_achieved=0
while IFS="=" read key value ; do
process_entry key value && one_is_achieved=1
done <<<$(output_entries)
# Dealing with one_is_achieved
[...]
the function output_entries() outputs key-values lines:
k1=some stuff
k2=other stuff
k3=remaining stuff
Problem, the command subtitution $() captures stdout but replace ends of line by spaces, causing k1 entry to be some stuff k2=other stuff k3= remaining stuff.
Is there a way to prevent the replacement or to restore the ends of lines?
Note: I solved my problem this way:
while IFS="=" read key value ; do
process_entry key value && one_is_achieved=1
done < <(output_entries)
But I think the original question is still relevant (preserving or restoring end of lines).
It is how command substitution is implemented in bash shell. The $(..) in unquoted form removes any embedded newlines because of the default word-splitting done by the shell. The default IFS value is $' \t\n' and unquoted expansion causes the splitting to happen based on any of these 3 characters.
You need to quote the substitution to prevent the word-splitting from happening. From the GNU bash man page or reset the IFS value by setting IFS="" before the substitution is expanded.
$(command)
Bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
This question already has answers here:
Honoring quotes while reading shell arguments from a file
(1 answer)
Reading quoted/escaped arguments correctly from a string
(4 answers)
Closed 5 years ago.
As show below, command-substitution changes the interpretation of quoted command-line arguments. What is going on under the hood, and is there a workaround?
$ cat dumpargs.sh
#! /usr/bin/env bash
for i in "$#"
{
echo "$i"
}
$ cat testfile.txt
'1 space' '2 space'
$ ./dumpargs.sh $(cat testfile.txt) ## produced undesired output
'1
space'
'2
space'
$ ./dumpargs.sh '1 space' '2 space' ## produces desired output
1 space
2 space
When you write ./dumpargs.sh '1 space' '2 space' on the command line, the shell interprets the single-quotes before passing arguments to the script. Argument #1 will have the value 1 space, argument #2 will ahve the value 2 space. The single-quotes are not part of the values.
When you write ./dumpargs.sh $(cat testfile.txt),
the shell will not try to interpret the content of testfile.txt.
The shell only interprets the actual text entered on the command line.
Substituted content like this example, or for example values of variables are not interpreted, but used literally.
Finally, the shell performs word splitting on the string on the command line, using the delimiters in IFS.
So the single-quotes will be used literally,
and the content is simply split by whitespace.
One possible workaround is to store one argument per line, without single-quotes, and make dumpargs.sh take the arguments from standard output:
while read -r line; do
echo "$line"
done
You can call it with ./dumpargs.sh < testfile.txt.
This question already has answers here:
Capturing multiple line output into a Bash variable
(7 answers)
Closed 6 years ago.
I know $(cmd) captures the output of cmd into a string. However, it doesn't take care of the newline characters at the end.
Here's a demonstration:
a=$(echo x)
b=$(echo -n x)
[ "$a" = "$b" ] && echo equal
a and b capture outputs differed by a "\n" but these variables have the same value.
So my goal is still to capture the output of a command, but this time I want to preserve the last newline character(s) if there are any.
Trailing newlines are removed
POSIX requires that the $(…) notation (or the equivalent using back-ticks) strips all trailing newlines from the end of the string that is captured.
§6.2.3 Command substitution
…, removing sequences of one or more <newline> characters at the end of the substitution.
There isn't a simple way around that, or to detect how many newlines were deleted.
Add a single newline to the end of the output
If you have Bash 4.x (4.3 tested) you can play with shell parameter expansion and the substring notation, and add a dummy line of output to the end of the original string (that's the echo n in this example):
$ x=$(echo pandemonium; blanklines 4; echo n)
$ echo "$x"
pandemonium
n
$ y=${x: 0: -1}
$ echo "$y"
pandemonium
$
When using Bash 3.2 (GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)) on macOS Sierra, the expansion for y generates an error:
$ y=${x: 0: -1}
-bash: -1: substring expression < 0
$
See also Capturing multiple line output to a Bash variable.
Reliable and POSIX-compliant
Or you can use a simpler, more portable (POSIX-compatible) substitution suggested by the answer — as applied to this answer:
$ y=${x%n}
$ echo "$y"
pandemonium
$
Given that this works in strict POSIX shells and both Bash 3.x and 4.x, there's no need for the substring variant.
What does the triple-less-than-sign bash operator, <<<, mean, as inside the following code block?
LINE="7.6.5.4"
IFS=. read -a ARRAY <<< "$LINE"
echo "$IFS"
echo "${ARRAY[#]}"
Also, why does $IFS remain to be a space, not a period?
It redirects the string to stdin of the command.
Variables assigned directly before the command in this way only take effect for the command process; the shell remains untouched.
From man bash
Here Strings
A variant of here documents, the format is:
<<<word
The word is expanded and supplied to the command on its standard input.
The . on the IFS line is equivalent to source in bash.
Update: More from man bash (Thanks gsklee, sehe)
IFS The Internal Field Separator that is used for word splitting
after expansion and to split lines into words with the read
builtin command. The default value is "<space><tab><new‐line>".
yet more from man bash
The environment for any simple command or function may be augmented
temporarily by prefixing it with parameter assignments, as described
above in PARAMETERS. These assignment statements affect only the environment seen by that command.
The reason that IFS is not being set is that bash isn't seeing that as a separate command... you need to put a line feed or a semicolon after the command in order to terminate it:
$ cat /tmp/ifs.sh
LINE="7.6.5.4"
IFS='.' read -a ARRAY <<< "$LINE"
echo "$IFS"
echo "${ARRAY[#]}"
$ bash /tmp/ifs.sh
7 6 5 4
but
$ cat /tmp/ifs.sh
LINE="7.6.5.4"
IFS='.'; read -a ARRAY <<< "$LINE"
echo "$IFS"
echo "${ARRAY[#]}"
$ bash /tmp/ifs.sh
.
7 6 5 4
I'm not sure why doing it the first way wasn't a syntax error though.
This question already has answers here:
Command to escape a string in bash
(5 answers)
Closed 4 years ago.
Is there a command that not just echos it's argument but also escapes them if needed (e.g. if a argument contains white space or a special character)?
I'd need it in some shell magic where instead of executing a command in one script I echo the command. This output gets piped to a python script that finally executes the commands in a more efficient manner (it loads the main() method of the actual target python script and executes it with the given arguments and an additional parameter by witch calculated data is cached between runs of main()).
Instead of that I could of course port all the shell magic to python where I wouldn't need to pipe anything.
With bash, the printf builtin has an additional format specifier %q, which prints the corresponding argument in a friendly way:
In addition to the standard printf(1) formats, %b causes printf to expand backslash escape sequences in the corresponding argument (except that \c terminates output, backslashes in \', \", and \? are not removed, and octal escapes beginning with \0 may contain up to four digits), and %q causes printf to output the corresponding argument in a format that can be reused as shell input.
So you can do something like this:
printf %q "$VARIABLE"
printf %q "$(my_command)"
to get the contents of a variable or a command's output in a format which is safe to pass in as input again (i.e. spaces escaped). For example:
$ printf "%q\n" "foo bar"
foo\ bar
(I added a newline just so it'll be pretty in an interactive shell.)