Command-substitution breaks quoted arguments with spaces [duplicate] - bash

This question already has answers here:
Honoring quotes while reading shell arguments from a file
(1 answer)
Reading quoted/escaped arguments correctly from a string
(4 answers)
Closed 5 years ago.
As show below, command-substitution changes the interpretation of quoted command-line arguments. What is going on under the hood, and is there a workaround?
$ cat dumpargs.sh
#! /usr/bin/env bash
for i in "$#"
{
echo "$i"
}
$ cat testfile.txt
'1 space' '2 space'
$ ./dumpargs.sh $(cat testfile.txt) ## produced undesired output
'1
space'
'2
space'
$ ./dumpargs.sh '1 space' '2 space' ## produces desired output
1 space
2 space

When you write ./dumpargs.sh '1 space' '2 space' on the command line, the shell interprets the single-quotes before passing arguments to the script. Argument #1 will have the value 1 space, argument #2 will ahve the value 2 space. The single-quotes are not part of the values.
When you write ./dumpargs.sh $(cat testfile.txt),
the shell will not try to interpret the content of testfile.txt.
The shell only interprets the actual text entered on the command line.
Substituted content like this example, or for example values of variables are not interpreted, but used literally.
Finally, the shell performs word splitting on the string on the command line, using the delimiters in IFS.
So the single-quotes will be used literally,
and the content is simply split by whitespace.
One possible workaround is to store one argument per line, without single-quotes, and make dumpargs.sh take the arguments from standard output:
while read -r line; do
echo "$line"
done
You can call it with ./dumpargs.sh < testfile.txt.

Related

print newline in process substitution [duplicate]

This question already has answers here:
Printf example in bash does not create a newline
(9 answers)
Closed 3 years ago.
This is in bash(5.0.3(1)-release) / ubuntu 19
When I ran :
printf "%s" $'\n'
It printed out a new line.
Now when I run:
result="$(printf "%s" $'\n')"
printf "<%s>" "$result"
I expect $result contains a newline, but it's empty.
Can someone explain ?
From posix shell manual, from chapter about process substitution, emphasis mine:
The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more <newline>s at the end of the substitution.
It's impossible to store trailing newlines with process substitution. You can use for example printf -v bash extension:
printf -v result "%s" $'\n'
For most portability, I go with encoding the string in hex using od or xxd and store the string in hex. Or just store the resulting string in a file.

bash: treat all arguments as a single string param [duplicate]

This question already has answers here:
Stop shell wildcard character expansion?
(4 answers)
Closed 3 years ago.
This will be better explained with an example. I'm building a little bash script that reruns the passed commandline every 1 second. this is my first attempt:
-- file wtbash
watch -n1 "$#"
If I issue:
wt "df -h | grep sda"
It works ok. I want to be able to do:
wt df -h | grep sda
And to work just the same
That is, I want to treat "df -h | grep sda" as a single string param
There's an app a variable for that! From man bash:
* Expands to the positional parameters, starting from one. When the expan‐
sion is not within double quotes, each positional parameter expands to a
separate word. In contexts where it is performed, those words are sub‐
ject to further word splitting and pathname expansion. When the expan‐
sion occurs within double quotes, it expands to a single word with the
value of each parameter separated by the first character of the IFS spe‐
cial variable. That is, "$*" is equivalent to "$1c$2c...", where c is
the first character of the value of the IFS variable. If IFS is unset,
the parameters are separated by spaces. If IFS is null, the parameters
are joined without intervening separators.
So change your script to:
#!/bin/bash
watch -n1 "$*"
But that won't work for pipes. You will still need to escape those:
wt df -h \| grep sda

Take care of last newline while capturing commands output [duplicate]

This question already has answers here:
Capturing multiple line output into a Bash variable
(7 answers)
Closed 6 years ago.
I know $(cmd) captures the output of cmd into a string. However, it doesn't take care of the newline characters at the end.
Here's a demonstration:
a=$(echo x)
b=$(echo -n x)
[ "$a" = "$b" ] && echo equal
a and b capture outputs differed by a "\n" but these variables have the same value.
So my goal is still to capture the output of a command, but this time I want to preserve the last newline character(s) if there are any.
Trailing newlines are removed
POSIX requires that the $(…) notation (or the equivalent using back-ticks) strips all trailing newlines from the end of the string that is captured.
§6.2.3 Command substitution
…, removing sequences of one or more <newline> characters at the end of the substitution.
There isn't a simple way around that, or to detect how many newlines were deleted.
Add a single newline to the end of the output
If you have Bash 4.x (4.3 tested) you can play with shell parameter expansion and the substring notation, and add a dummy line of output to the end of the original string (that's the echo n in this example):
$ x=$(echo pandemonium; blanklines 4; echo n)
$ echo "$x"
pandemonium
n
$ y=${x: 0: -1}
$ echo "$y"
pandemonium
$
When using Bash 3.2 (GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)) on macOS Sierra, the expansion for y generates an error:
$ y=${x: 0: -1}
-bash: -1: substring expression < 0
$
See also Capturing multiple line output to a Bash variable.
Reliable and POSIX-compliant
Or you can use a simpler, more portable (POSIX-compatible) substitution suggested by the answer — as applied to this answer:
$ y=${x%n}
$ echo "$y"
pandemonium
$
Given that this works in strict POSIX shells and both Bash 3.x and 4.x, there's no need for the substring variant.

What does the Bash operator <<< (i.e. triple less than sign) mean?

What does the triple-less-than-sign bash operator, <<<, mean, as inside the following code block?
LINE="7.6.5.4"
IFS=. read -a ARRAY <<< "$LINE"
echo "$IFS"
echo "${ARRAY[#]}"
Also, why does $IFS remain to be a space, not a period?
It redirects the string to stdin of the command.
Variables assigned directly before the command in this way only take effect for the command process; the shell remains untouched.
From man bash
Here Strings
A variant of here documents, the format is:
<<<word
The word is expanded and supplied to the command on its standard input.
The . on the IFS line is equivalent to source in bash.
Update: More from man bash (Thanks gsklee, sehe)
IFS The Internal Field Separator that is used for word splitting
after expansion and to split lines into words with the read
builtin command. The default value is "<space><tab><new‐line>".
yet more from man bash
The environment for any simple command or function may be augmented
temporarily by prefixing it with parameter assignments, as described
above in PARAMETERS. These assignment statements affect only the environment seen by that command.
The reason that IFS is not being set is that bash isn't seeing that as a separate command... you need to put a line feed or a semicolon after the command in order to terminate it:
$ cat /tmp/ifs.sh
LINE="7.6.5.4"
IFS='.' read -a ARRAY <<< "$LINE"
echo "$IFS"
echo "${ARRAY[#]}"
$ bash /tmp/ifs.sh
7 6 5 4
but
$ cat /tmp/ifs.sh
LINE="7.6.5.4"
IFS='.'; read -a ARRAY <<< "$LINE"
echo "$IFS"
echo "${ARRAY[#]}"
$ bash /tmp/ifs.sh
.
7 6 5 4
I'm not sure why doing it the first way wasn't a syntax error though.

echo that shell-escapes arguments [duplicate]

This question already has answers here:
Command to escape a string in bash
(5 answers)
Closed 4 years ago.
Is there a command that not just echos it's argument but also escapes them if needed (e.g. if a argument contains white space or a special character)?
I'd need it in some shell magic where instead of executing a command in one script I echo the command. This output gets piped to a python script that finally executes the commands in a more efficient manner (it loads the main() method of the actual target python script and executes it with the given arguments and an additional parameter by witch calculated data is cached between runs of main()).
Instead of that I could of course port all the shell magic to python where I wouldn't need to pipe anything.
With bash, the printf builtin has an additional format specifier %q, which prints the corresponding argument in a friendly way:
In addition to the standard printf(1) formats, %b causes printf to expand backslash escape sequences in the corresponding argument (except that \c terminates output, backslashes in \', \", and \? are not removed, and octal escapes beginning with \0 may contain up to four digits), and %q causes printf to output the corresponding argument in a format that can be reused as shell input.
So you can do something like this:
printf %q "$VARIABLE"
printf %q "$(my_command)"
to get the contents of a variable or a command's output in a format which is safe to pass in as input again (i.e. spaces escaped). For example:
$ printf "%q\n" "foo bar"
foo\ bar
(I added a newline just so it'll be pretty in an interactive shell.)

Resources