Bash concatenating lines in a variable [duplicate] - bash

This question already has an answer here:
Variable expansion is different in zsh from that in bash
(1 answer)
Closed 7 years ago.
I was trying to convert my zsh prompt to bash and I had this line:
local gitstat=$(git status 2>/dev/null | grep '^\(Untracked\|Changes\|Changed but not updated:\)')
In zsh, when in a Git repository with unstaged changes and untracked files, the variable $gitstat is set to
Changes not staged for commit:
Untracked files:
but in bash, it is set to
Changes not staged for commit: Untracked files:
It seems that bash is concatenating the lines together when run in a subshell and assigning to a variable, and I can't find any explanation why or how to stop this.

bash isn't concatenating the lines; rather, you are not quoting the expansion of $gitstatus before printing it. If you write
echo $gitstatus
bash first expands $gitstatus, then performs word-splitting on the resulting string. Any whitespace, including embedded newlines, are treated as word separators, so that the string is split into multiple words, each of which is passed to echo as a separate argument. echo then displays each argument on one line, separated by a single space.
In contrast, quoting the expansion
echo "$gitstatus"
prevents the argument from being split into words, and the entire string (embedded newlines and all) are passed to echo as a single argument.
The difference between bash and zsh in this case is that zsh, in contrast to the POSIX specification, does not perform word-splitting on unquoted parameter expansions.

Related

Bash - how to evaluate curly brackets wildcard from a variable [duplicate]

This question already has answers here:
Brace expansion with variable? [duplicate]
(6 answers)
Closed 4 years ago.
Consider the following script:
#! /bin/bash -e
echo {foo,bar}
EX={foo,bar}
echo ${EX}
The output of this script is:
foo bar
{foo,bar}
I would like the the echo command to perform brace expansion on ${EX}. Thus, I would like to see an output of
foo bar
foo bar
I want to create a script where the user can supply a path with curly brackets where every expanded version of it is copied.
Something like this:
#! /bin/bash -e
$SOURCES=$1
$TARGET=$2
cp -r ${SOURCES} ${TARGET}
How can I achieve this?
This is a way:
ex=({foo,bar,baz})
echo ${ex[#]}
foo bar baz
See man bash:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
As you see, variable expansion happens later than brace expansion.
Fortunately, you don't need it at all: let the user specify the braced paths, let the shell expand them. You can then just
mv "$#"
If you need to separate the arguments, use an array and parameter expansion:
sources=("${#:1:$#-1}")
target=${#: -1}
mv "${sources[#]}" "$target"
Brace expansion does not work the way you are attempting to use it. Brace expansion is basically used to generate lists to be applied within the context of the present command. You have two primary modes where brace expansion is used directly (and many more where brace expansion is used a part of another operator.) The two direct uses are to expand a list of items within a comma-separated pair of braces. e.g.
$ touch file_{a,b,c,d}.txt
After executing the command, brace expansion creates all four files with properly formatted file names in the present directory:
$ ls -1 file*.txt
file_a.txt
file_b.txt
file_c.txt
file_d.txt
You may also use brace-expansion in a similar manner to generate lists for loop iteration (or wherever a generated system/range of numbers in needed). The syntax for using brace expansion here is similar, but with .. delimiters within the braces (instead of ',' separation). The syntax is {begin..end..increment} (whereincrement can be both positive and negative) e.g.
$ for i in {20..-20..-4}; do echo $i; done)
20
16
12
8
4
0
-4
-8
-12
-16
-20
(note: using variables for begin, end or increment is not allowed without some horrible eval trickery -- avoid it.).

Processing output command with new lines redirected as Herestring [duplicate]

This question already has answers here:
How to avoid bash command substitution to remove the newline character?
(3 answers)
Closed 3 years ago.
I'm dealing with a while loop processing a Herestring input:
one_is_achieved=0
while IFS="=" read key value ; do
process_entry key value && one_is_achieved=1
done <<<$(output_entries)
# Dealing with one_is_achieved
[...]
the function output_entries() outputs key-values lines:
k1=some stuff
k2=other stuff
k3=remaining stuff
Problem, the command subtitution $() captures stdout but replace ends of line by spaces, causing k1 entry to be some stuff k2=other stuff k3= remaining stuff.
Is there a way to prevent the replacement or to restore the ends of lines?
Note: I solved my problem this way:
while IFS="=" read key value ; do
process_entry key value && one_is_achieved=1
done < <(output_entries)
But I think the original question is still relevant (preserving or restoring end of lines).
It is how command substitution is implemented in bash shell. The $(..) in unquoted form removes any embedded newlines because of the default word-splitting done by the shell. The default IFS value is $' \t\n' and unquoted expansion causes the splitting to happen based on any of these 3 characters.
You need to quote the substitution to prevent the word-splitting from happening. From the GNU bash man page or reset the IFS value by setting IFS="" before the substitution is expanded.
$(command)
Bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.

Can I get the original quote characters surrounding arguments in my Bash script? [duplicate]

This question already has answers here:
How can I preserve quotes in printing a bash script's arguments
(7 answers)
Closed 4 years ago.
I have a script that logs the user argument list. This list is later processed by getopt.
If the script is started like:
./script.sh -a 'this is a sentence' -b 1
... and then I save "$#", I get:
-a this is a sentence -b 1
... without the single quotes. I think (because of the way Bash treats quotes) these are removed and are not available to the script.
For logging accuracy, I'd like to include the quotes too.
Can the original argument list be obtained without needing to quote-the-quotes?
No, there is no way to obtain the command line from before the shell performed whitespace tokenization, wildcard expansion, and quote removal on it.
If you want to pass in literal quotes, try
./script.sh '"-a"' '"this is a sentence"' '"-b"' '"1"'
Notice also how your original command line could have been written
'./script.sh' '-a' 'this is a sentence' '-b' '1'

How can I concatenate a var with curly brackets in bash [duplicate]

This question already has answers here:
Brace expansion with variable? [duplicate]
(6 answers)
Closed 4 years ago.
Consider the following script:
#! /bin/bash -e
echo {foo,bar}
EX={foo,bar}
echo ${EX}
The output of this script is:
foo bar
{foo,bar}
I would like the the echo command to perform brace expansion on ${EX}. Thus, I would like to see an output of
foo bar
foo bar
I want to create a script where the user can supply a path with curly brackets where every expanded version of it is copied.
Something like this:
#! /bin/bash -e
$SOURCES=$1
$TARGET=$2
cp -r ${SOURCES} ${TARGET}
How can I achieve this?
This is a way:
ex=({foo,bar,baz})
echo ${ex[#]}
foo bar baz
See man bash:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
As you see, variable expansion happens later than brace expansion.
Fortunately, you don't need it at all: let the user specify the braced paths, let the shell expand them. You can then just
mv "$#"
If you need to separate the arguments, use an array and parameter expansion:
sources=("${#:1:$#-1}")
target=${#: -1}
mv "${sources[#]}" "$target"
Brace expansion does not work the way you are attempting to use it. Brace expansion is basically used to generate lists to be applied within the context of the present command. You have two primary modes where brace expansion is used directly (and many more where brace expansion is used a part of another operator.) The two direct uses are to expand a list of items within a comma-separated pair of braces. e.g.
$ touch file_{a,b,c,d}.txt
After executing the command, brace expansion creates all four files with properly formatted file names in the present directory:
$ ls -1 file*.txt
file_a.txt
file_b.txt
file_c.txt
file_d.txt
You may also use brace-expansion in a similar manner to generate lists for loop iteration (or wherever a generated system/range of numbers in needed). The syntax for using brace expansion here is similar, but with .. delimiters within the braces (instead of ',' separation). The syntax is {begin..end..increment} (whereincrement can be both positive and negative) e.g.
$ for i in {20..-20..-4}; do echo $i; done)
20
16
12
8
4
0
-4
-8
-12
-16
-20
(note: using variables for begin, end or increment is not allowed without some horrible eval trickery -- avoid it.).

How do I store a command in a variable and use it in a pipeline? [duplicate]

This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 6 years ago.
If i use this command in pipeline, it's working very well;
pipeline ... | grep -P '^[^\s]*\s3\s'
But if I want to set grep into variable like:
var="grep -P '^[^\s]*\s3\s'"
And if I put variable in pipeline;
pipeline ... | $var
nothing happens, like there isn't any matches.
Any help what am I doing wrong?
The robust way to store a simple command in a variable in Bash is to use an array:
# Store the command names and arguments individually
# in the elements of an *array*.
cmd=( grep -P '^[^\s]*\s3\s' )
# Use the entire array as the command to execute - be sure to
# double-quote ${cmd[#]}.
echo 'before 3 after' | "${cmd[#]}"
If, by contrast, your command is more than a simple command and, for instance, involves pipes, multiple commands, loops, ..., defining a function is the right approach:
# Define a function encapsulating the command...
myGrep() { grep -P '^[^\s]*\s3\s'; }
# ... and use it:
echo 'before 3 after' | myGrep
Why what you tried didn't work:
var="grep -P '^[^\s]*\s3\s'"
causes the single quotes around the regex to become a literal, embedded part of $var's value.
When you then use $var - unquoted - as a command, the following happens:
Bash performs word-splitting, which means that it breaks the value of $var into words (separate tokens) by whitespace (the chars. defined in special variable $IFS, which contains a space, a tab, and a newline character by default).
Bash also performs globbing (pathname expansion) on the resulting works, which is not a problem here, but can have unintended consequences in general.
Also, if any of your original arguments had embedded whitespace, word splitting would split them into multiple words, and your original argument partitioning is lost.
(As an aside: "$var" - i.e., double-quoting the variable reference - is not a solution, because then the entire string is treated as the command name.)
Specifically, the resulting words are:
grep
-P
'^[^\s]*\s3\s' - including the surrounding single quotes
The words are then interpreted as the name of the command and its arguments, and invoked as such.
Given that the pattern argument passed to grep starts with a literal single quote, matching won't work as intended.
Short of using eval "$var" - which is NOT recommended for security reasons - you cannot persuade Bash to see the embedded single quotes as syntactical elements that should be removed (a process appropriate called quote removal).
Using an array bypasses all these problems by storing arguments in individual elements and letting Bash robustly assemble them into a command with "${cmd[#]}".
What you are doing wrong is trying to store a command in a variable. For simplicity, robustness, etc. commands are stored in aliases (if no arguments) or functions (if arguments), not variables. In this case:
$ alias foo='grep X'
$ echo "aXb" | foo
aXb
I recommend you read the book Shell Scripting Recipes by Chris Johnson ASAP to get the basics of shell programming and then Effective Awk Programming, 4th Edition, by Arnold Robbins when you're ready to start writing scripts to manipulate text.

Resources