I want to decide whether to always omit quotes for variables appearing withing a Bash [[ test. I interpret the man page to say that it is permissible to do so without any loss of correctness.
I devised this simplistic "test" to verify my thinking and the "expected behaviour" but it may prove absolutely nothing, take a look at it:
x='1 == 2 &&'
if [[ $x == '1 == 2 &&' ]]; then
echo yes
else
echo no
fi
Note I am not writing this as such:
x='1 == 2 &&'
if [[ "$x" == '1 == 2 &&' ]]; then
echo yes
else
echo no
fi
which so far has always been my style, for consistency if nothing else.
Is is safe to switch my coding convention to always omit quotes for variables appearing within [[ tests?
I am trying to learn Bash and I am trying to do so picking up good habits, good style and correctness..
The key thing to remember is that quotes and escaping within pattern matching contexts always cause their contents to become literal. Quoting on the left hand side of an == within [[ is never necessary, only the right side is interpreted as a pattern. Quoting on the right hand side is necessary if you want a literal match and to avoid interpretation of pattern metacharacters within the variable.
In other words, [ "$x" = "$x" ] and [[ $x == "$x" ]] are mostly equivalent, and of course in Bash the latter should be preferred.
One quick tip: think of the operators of the [[ ]] compound comand as being the same grammar-wise as other control operators such as elif, do, ;;, and ;;& (though technically in the manual they're in their own category). They're really delimiters of sections of a compound command, which is how they achieve seemingly magical properties like the ability to short-circuit expansions. This should help to clarify a lot of the behavior of [[, and why it's distinct from e.g. the arithmetic operators, which are not like that.
More examples: http://mywiki.wooledge.org/BashFAQ/031#Theory
No. You should not get in the habit of always omitting quotes, even if they appear within [[ tests. Bash is famous for burning people for leaving off quotes :-)
In bash the [[ ]] should always evaluate as an expression, so the script will continue to function. The risk is that a logic error may pop up unnoticed. In all cases that I can think of off the top of my head it would be fine. However, quotes allow you to be specific about what you want, and are also self-documenting in addition to being safer.
Consider this expression:
if [[ "$INT" =~ ^-?[0-9]+$ ]]; then
It would still work without the quotes because it is in between [[ ]], but the quotes are clarifying and do not cause any issues.
At any rate, this is my opinion as a guy who has received a royal hosing at the hand of Bash because I failed to put " " around something that needed them :'(
My bash hacking friend once said, "use quotes liberally in Bash." That advice has served me well.
Related
I apologize beforehand for this question, which is probably both ill formulated and answered a thousand times over. I get the feeling that my inability to find an answer is that I don't quite know how to ask the question.
I'm writing a script that traverses folders in a bunch of mounted external hard drives, like so:
for g in /Volumes/compartment-?/{Private/Daniel,Daniel}/Projects/*/*
It then proceeds to perform long-running tasks on each of the directories found there. Because these operations are io-intensive rather than cpu-intensive, I thought I'd add the option to provide which "compartment" I want to work in, so that I can parallelize the workloads.
But, doing
cmp="?"
[[ ! "$1" = "" ]] && cmp="$1"
And then,
for g in /Volumes/compartment-$cmp/{Private/Daniel,Daniel}/Projects/*/*
Doesn't work - the question mark that should expand to all compartments instead becomes literal, so I get an error that "compartment-?" doesn't exist, which is of course true.
How do I create a variable with a value that "expands," like dir="./*" working with ls $dir?
EDIT: Thanks to #dan for the answer. I was brought up to be courteous and thank people, so I did thank him for it in a comment on his question, but that comment has been removed, and I'm anxious that repeating it might be some kind of infraction here. I ended up simply escaping my question mark glob character, i.e. \?, since for this script I only need to either search all drives or one particular drive. But I'll keep the answer handy for the next time I write a script where I'd like to support more advanced arguments.
Brace expansion occurs before variable expansion. Pathname/glob expansion (eg ?, *) occurs last. Therefore you can't use the glob character ? in a variable, and in a brace expansion.
You can use a glob expression in an unquoted variable, without brace expansion. Eg. q=\?; echo compartment-$q is equivalent to echo compartment-?.
To solve your problem, you could define an array based on the input argument:
if [[ $1 ]]; then
[[ -d /Volumes/compartment-$1 ]] || exit 1
files=("/Volumes/compartment-$1"/{Private/Daniel,Daniel}/Projects/*/*)
else
files=(/Volumes/compartment-?/{Private/Daniel,Daniel}/Projects/*/*)
fi
# then iterate the list:
for i in "${files[#]}"; do
...
Another option is a nested loop. The path expression in the outer loop doesn't use brace expansion, so (unlike the first example) it can expand a glob in $1 (or default to ? if $1 is empty):
for i in /Volumes/compartments-${1:-?}; do
[[ -d $i ]] &&
for j in {Private/Daniel,Daniel}/Projects/*/*; do
[[ -e $j ]] || continue
...
Note that the second example expands a glob expression passed in $1 (eg. ./script '[1-9]'). The first example does not.
Remember that pathname expansion has the property of expanding only to existing files, or literally. shopt -s nullglob guarantees expansion only to existing files (or nothing).
You should either use nullglob, or check that each file or directory exists, like in the examples above.
Using $1 unquoted also subjects it to word splitting on whitespace. You can set IFS= (empty) to avoid this.
Run the following in bash:
stuff=`rpm -ql <some package> | grep dasdasdfd`
(non existent file in package, exit code = 1, stdout is empty)
if [ -f $stuff ]; then echo "whaaat"; fi
Above command checks if file exists... but:
file $stuff
Just prints usage info for file... and
stat $stuff
Missing operand...
Can someone please explain why? Is this a bug? Am I doing something wrong? I just want to make sure that a file that's in the package is present on fs
You probably need to surround $stuff in quotes
if [ -f "$stuff" ]; then
As a general rule, you almost always want to add quotes around pathnames everywhere you use them.
I find it more useful to think or variables in shell scripting as "macros", which are expanded on first use to their value. This is different from variables in almost every other programming language.
So if $stuff contains hello world (notice the space), it would be the same as if you've typed:
[ -f hello world ]
which is obviously an error.
In this case, you mentioned that you're dealing with a non-existent file, so $stuff is actually empty, which would be like typing:
[ -f ]
Which is actually valid, but always succeeds. This is a bit of obscure test behaviour, from the POSIX spec we read that test always succeeds if there if only a single argument (in this case, the argument is -f):
1 argument:
Exit true (0) if $1 is not null; otherwise, exit false.
This is probably to facilitate the writing of:
[ $variable_that_may_or_may_not_be_defined ]
If you add quotes, you're passing 2 arguments, and more sane things happen:
if [ -f "" ]; then
Martin Tournoij's answer and DevSolar's answer both provide correct solutions and helpful background info: with respect to [ ... ] in one case, and [[ ... ]] in the other.
Since it may not be obvious if and when to choose [[ ... ]] over [ ... ] (and its (virtual) alias, test ...), let me attempt a summary:
If your code must be portable (POSIX-compliant), you MUST use [ ... ] (or test ...).
Tokens inside [ ... ] are parsed just like arguments passed to an executable, so you must double-quote your variable references, unless you explicitly want all shell expansions - notably word splitting (automatic splitting into multiple tokens by whitespace) and globbing - applied to them.
[ -f "$stuff" ] # double-quoting required, if $stuff has embedded whitespace
If you know that your code will be run with bash, you can use [[ ... ]] for more features and fewer surprises.
Tokens inside [[ ... ]] are parsed in a special context in which neither word splitting nor pathname expansion (globbing) are applied (though other expansions, such as parameter expansion, do occur), so there is typically no need to double-quote variable references.
[[ -f $stuff ]] # double-quoting optional
Note that ksh and zsh also support [[ ... ]] (presumably with subtle variations in behavior).
For more background info, such as the additional features that [[ ... ]] offers, read on.
[[ ... ]] improves on [ ... ] / test ... as follows:
"RHS" below means "right-hand side", i.e., the right operand of a binary operator.
(typically) requires NO quoting of variable references (except on the RHS of == and =~ to specify a literal string or substring(s))
f='some file'; [[ -f $f ]] # ok, double quotes optional
v='*'; [[ $v == '*' ]] # ok, double quotes optional
Neither word splitting nor pathname expansion is applied inside [[ ... ]], so it's safe to use unquoted references to variables whose values have embedded whitespace and/or values such as * that would normally lead to globbing.
offers string pattern matching with = / ==, with an unquoted pattern on the RHS (or at least unquoted pattern metachars.)
[[ abc == a* ]] && echo yes # matches; use of = instead of == works too
Caveat: Thus, on the RHS of = / == you must double-quote variable references (or single-quote literals) if you want their values to be treated as literals.
v='a*'; [[ abc == "$v" ]] # does NOT match
offers regex matching with =~, with an unquoted extended regular expression on the RHS (or at least unquoted regex metachars.)
[[ abc =~ ^a.+$ ]] && echo yes # matches
Caveat: Thus, on the RHS of =~ you must double-quote variable references (or single-quote literals) if you want their values to be treated as literals.
v='a.+'; [[ abc =~ ^"$v"$ ]] # does NOT match
Also note that the unquoted / quoted distinction was only introduced in bash 3.2 - you can still use shopt -s compat31 to have single- and double-quoted strings treated as regexes, too.
Caveat: The regex dialect understood by =~ is platform-specific, so a regex that works on one platform may not work on another (this is one of the few cases where bash's behavior is platform-dependent). For instance, on Linux you can use \b and \< / \> for word-boundary assertions, whereas BSD/macOS only supports [[:<]] and [[:>]], which, in turn, Linux doesn't support - see this answer of mine.
offers grouping and negation with unescaped (, ), and ! chars.
offers use of && and || (Boolean AND and OR)
[[ (3 -gt 2) && ! -f / ]] && echo yes
Note that, inside [[ ... ]], && has higher precedence than || - unlike OUTSIDE (as so-called [command-]list operators, where they combine entire commands / command lists), where they have equal precedence.
(while [ and test have -a and -o, even the POSIX spec. for test cautions against their use)
within [[ ... ]], you may spread your conditional across multiple lines for readability without the need for the line-continuation char. (\), assuming the line breaks come after && or ||, as codeforester points out.
[[ ... ]] is faster than [ ... ], though that will typically not matter.
If you are interested in relative performance, see this answer of mine.
Implementation notes re [ and test:
[[ a is shell keyword (supported in bash, ksh, and zsh), which allows for different parsing rules, as described above.
By contrast, [ and test are builtins in all major POSIX-like shells (bash, ksh, zsh, dash).
In addition, both [ and test exist as external utilities (executable files that require a separate process to invoke), as mandated by POSIX.
In fact, you need external utility versions so as to be able to use [ or test in "shell-less" invocation scenarios such as when passing a test to find -exec or xargs.
While the [ utility could conceivably be implemented as a symlink to the test utility (as long as test knows how it was invoked and enforces the closing ] when invoked as [), in practice they are often (always?) separate executables (true on Linux and macOS / BSD, for instance; on Linux, their content differs, whereas on macOS / BSD their content is identical (they are copies of the same file)).
One option would be to put $stuff in quotes, as Carpetsmoker said.
But since this is tagged bash, and because catering for whitespace in filenames is a pain, you could go for:
if [[ -f $stuff ]]
As opposed to [ which is an alias for test, the [[ construct "knows" how to handle the contents of $stuff correctly.
Bash's manpage teaches that [[ == ]] matches patterns. In Bash therefore, why does the following not print matched?
Z=abc; [[ "$Z" == 'a*' ]] && echo 'matched'
The following however does indeed print matched:
Z=abc; [[ "$Z" == a* ]] && echo 'matched'
Isn't this exactly backward? Why does the a*, without the quotes, not immediately expand to list whatever filenames happen to begin with the letter a in the current directory? And besides, why doesn't the quoted 'a*' work in any case?
Glob pattern must not be quoted to make it work.
This should also work with just glob pattern out of quote whereas static text is still qupted:
[[ "$Z" == "a"* ]] && echo 'matched'
matched
[[ "$Z" == "ab"* ]] && echo 'matched'
matched
Explanation snippet from man page:
When the == and != operators are used, the string to the right of
the operator is considered a pattern and matched according to the
rules described below under Pattern Matching. If the shell option
nocasematch is enabled, the match is performed without regard to
the case of alphabetic characters. The return value is 0 if the
string matches (==) or does not match (!=) the pattern, and 1
otherwise. Any part of the pattern may be quoted to force it to be
matched as a string.
Additionally, one of the reasons to use [[ over [ is that [[ is a shell built-in and thus can have its own syntax and doesn't need to follow the normal expansion rules (which is why the arguments to [[ aren't subject to word-splitting for example).
While the existing answer is correct, I don't believe that it tells the full story.
Globs have two uses. There is a difference in behaviour between globs inside a [[ ]] construct which test the contents of a variable against a pattern and other globs, which expand to list a range of files. In either case, if you put quotes around character, it will be interpreted literally and not expanded.
It is also worth mentioning that the variable on the left hand side doesn't need to be quoted after the [[, so you could write your code like this:
Z=abc; [[ $Z == a* ]] && echo 'matched'
It is also possible to use a single = but the == looks more familiar to those coming from other coding backgrounds, so personally I prefer to use it in bash as well. As mentioned in the comments, the single = is the more widely compatible, as it is used to test string equality in all of POSIX-compliant shells, e.g. [ "$a" = "abc" ]. For this reason you may prefer to use it in bash as well.
As always, Greg's wiki contains some good information on the subject of pattern matching in bash.
This question already has answers here:
Are double square brackets [[ ]] preferable over single square brackets [ ] in Bash?
(10 answers)
Closed 4 years ago.
I looked at bash man page and the [[ says it uses Conditional Expressions. Then I looked at Conditional Expressions section and it lists the same operators as test (and [).
So I wonder, what is the difference between [ and [[ in Bash?
[[ is bash's improvement to the [ command. It has several enhancements that make it a better choice if you write scripts that target bash. My favorites are:
It is a syntactical feature of the shell, so it has some special behavior that [ doesn't have. You no longer have to quote variables like mad because [[ handles empty strings and strings with whitespace more intuitively. For example, with [ you have to write
if [ -f "$file" ]
to correctly handle empty strings or file names with spaces in them. With [[ the quotes are unnecessary:
if [[ -f $file ]]
Because it is a syntactical feature, it lets you use && and || operators for boolean tests and < and > for string comparisons. [ cannot do this because it is a regular command and &&, ||, <, and > are not passed to regular commands as command-line arguments.
It has a wonderful =~ operator for doing regular expression matches. With [ you might write
if [ "$answer" = y -o "$answer" = yes ]
With [[ you can write this as
if [[ $answer =~ ^y(es)?$ ]]
It even lets you access the captured groups which it stores in BASH_REMATCH. For instance, ${BASH_REMATCH[1]} would be "es" if you typed a full "yes" above.
You get pattern matching aka globbing for free. Maybe you're less strict about how to type yes. Maybe you're okay if the user types y-anything. Got you covered:
if [[ $ANSWER = y* ]]
Keep in mind that it is a bash extension, so if you are writing sh-compatible scripts then you need to stick with [. Make sure you have the #!/bin/bash shebang line for your script if you use double brackets.
See also
Bash FAQ - "What is the difference between test, [ and [[ ?"
Bash Practices - Bash Tests
Server Fault - What is the difference between double and single brackets in bash?
[ is the same as the test builtin, and works like the test binary (man test)
works about the same as [ in all the other sh-based shells in many UNIX-like environments
only supports a single condition. Multiple tests with the bash && and || operators must be in separate brackets.
doesn't natively support a 'not' operator. To invert a condition, use a ! outside the first bracket to use the shell's facility for inverting command return values.
== and != are literal string comparisons
[[ is a bash
is bash-specific, though others shells may have implemented similar constructs. Don't expect it in an old-school UNIX sh.
== and != apply bash pattern matching rules, see "Pattern Matching" in man bash
has a =~ regex match operator
allows use of parentheses and the !, &&, and || logical operators within the brackets to combine subexpressions
Aside from that, they're pretty similar -- most individual tests work identically between them, things only get interesting when you need to combine different tests with logical AND/OR/NOT operations.
The most important difference will be the clarity of your code. Yes, yes, what's been said above is true, but [[ ]] brings your code in line with what you would expect in high level languages, especially in regards to AND (&&), OR (||), and NOT (!) operators. Thus, when you move between systems and languages you will be able to interpret script faster which makes your life easier. Get the nitty gritty from a good UNIX/Linux reference. You may find some of the nitty gritty to be useful in certain circumstances, but you will always appreciate clear code! Which script fragment would you rather read? Even out of context, the first choice is easier to read and understand.
if [[ -d $newDir && -n $(echo $newDir | grep "^${webRootParent}") && -n $(echo $newDir | grep '/$') ]]; then ...
or
if [ -d "$newDir" -a -n "$(echo "$newDir" | grep "^${webRootParent}")" -a -n "$(echo "$newDir" | grep '/$')" ]; then ...
In bash, contrary to [, [[ prevents word splitting of variable values.
How can you check which words is first alphabetically between two words?
For example in the code
#!/bin/bash
var1="apple"
var2="bye"
if [ $var1 \> $var2 ]
then
echo $var1
else
echo $var2
fi
I want it to print apple, since apple comes before bye alphabetically, but it isnt working as intended. What am I doing wrong?
What you need to do to solve the immediate problem is reverse the sense of your statement, since the "less than" operator is < rather than >.
Doing so will get it working correctly:
if [ $var1 \< $var2 ]
Alternatively, you can use the [[ variant which doesn't require the escaping:
if [[ $var1 < $var2 ]]
I prefer the latter because:
it looks nicer; and
the [[ variant is much more expressive and powerful.
You'll want to use the [[ ]] construct and print out the one that is less than the other
#!/bin/bash
var1="apple"
var2="bye"
if [[ $var1 < $var2 ]]; then
echo $var1
else
echo $var2
fi
It seems that you are suffering from two misconceptions about (bash) shell scripting.
First the line if [ $var1 > $var2 ] then echo ... is syntactically wrong and you should really paste your example commands or code rather than trying to re-type them from memory. It would be fine if you'd said if [[ "$var1" > "$var2" ]]; then ... or if [ "$var" \> "$var2" ]; then ....
Note that [[ is a bash specific conditional expression while [ (single bracket) introduces the shell built-in implementation of the /usr/bin/[ (alias for /usr/bin/test) command.
The old [ (test) command has much more constrained features than the [[ support in bash. It only support -lt, -eq ... and other integer comparisons and the various file and value length (-z' and-n) and other tests. It has no support for lexical/string or pattern (regex nor glob) comparisons. Thebashbuilt-in for[supports a number of the[[` extensions but, as shown some of them have to be explicitly escaped from the legacy parsing.
Also note that it's dangerous to use bar dereferences of $var (vs "$var" with the quotes). If the value assigned to var has any embedded spaces or various other operators which might be conflated with the switches to the test command.
Also you need the ; to separate the if command from the then clause.
Trying to write shell scripts as though shell were a normal programming language will lead you to quite a bit of this sort of confusion. Shells, such as bash have relatively little syntax and built-in functionality and most of that is glue around running commands. In early versions of UNIX the shell didn't have any built-in tests and relied entirely on the external test command. Over time more and more functionality was built-in the shell, often through aliases to those old commands (/usr/bin/[' is literally a link to the/usr/bin/testcommand and the shell built-ins for[andtest` are internal aliases to one another and implemented as (mostly?) compatible with the older (and still extant) external binaries.
Similarly all arithmetic operations in the early Bourne shells were done using external commands such as /usr/bin/expr. Korn shell and Bash added $((...)) and let and ((...)) expressions
for evaluating arithmetic expressions from within the shell without external command support.
Other examples relate to the support for arrays (declare) and parameter expansion ${var#...}
various other forms.
It's generally best to avoid most such features, or use them sparingly, as the resulting scripts because progressively less portable as you use them ... and the syntactic machinations rapidly overwhelm the code. At some point it's best to use Perl, Python, Ruby or some general purpose programming/scripting language for performing the general programming work and use shell for the purposes to which it was designed ... as glue around external commands, for marshaling data and variables into and out of those external commands/processes.