A few days ago, my lab's server suffered a serious meltdown when one of our interns accidentally copy-pasted this code on bash trying to delete node.js.
$ rm -rfv /usr/{bin/node,lib / node,share / man /* / node.*};
They tried a brace expansion to list up directories to delete, but notice spaces between the directory separators (/). This ended up deleting everything on our server because they applied sudo.
I tried this command on my virtual machine and confirmed that it was pretty much equivalent to rm -rf /.
I'm confused about the way bash interpreted the statement. When I try to create a simple, nondestructive command that does similar things (spaces working as separators for expansion,) I don't seem to be able to pull that off. I tried the first command but with for loop in bash:
$ for f in /usr/{bin/node,lib / node,share / man /* / node.*}; do echo $f; done
which listed some contents in node.js and the directories in /. This should confirm that this is not a special feature in rm.
But when I try something like this:
$ for f in {a,b c,d e}; echo $f
It results in a syntax error near echo where I expected a to e, each letter in a single line.
I did some research, but I couldn't find anything that explains this behavior.
Can someone please tell me, in the first command, how did bash interpret this command?
p.s. I found out that in zsh the 'for loop test' version doesn't work. Never tried the real 'rm test' though. I'm scared.
You must \-escape all spaces that are part of your brace expansion:
$ printf '%s\n' {a,b\ c}
a
b c
Brace expansions only work:
when they're neither neither single- nor double-quoted (you got that part right)
and when they're recognized as a single word (token) by the shell (that's where your attempt fell short) - hence the need for character-individual quoting of spaces with \.
Without this, bash breaks what you meant to be a brace expansion into multiple arguments, and brace expansion never happens - see below.
As for how bash parses /usr/{bin/node,lib / node,share / man /* / node.*}:
The following tells you the resulting arguments (with actual globbing omitted to better demonstrate what happens):
$1=/usr/{bin/node,lib
$2=/
$3=node,share
$4=/
$5=man
$6=/*
$7=/
$8=node.*}
As you can see:
The unescaped spaces caused the word-splitting to occur by spaces, breaking what you meant to be brace expressions into multiple arguments.
One of the resulting arguments /*, unfortunately, matches all (non-hidden) items in the root directory, and therefore wreaks havoc when passed to sudo rm.
Related
I have the following bash script:
$ echo $(dotnet run --project Updater)
UPDATE_NEEDED='0' MD5_SUM="7e3ad68397421276a205ac5810063e0a"
$ export UPDATE_NEEDED='0' MD5_SUM="7e3ad68397421276a205ac5810063e0a"
$ echo $UPDATE_NEEDED
0
$ export $(dotnet run --project Updater)
$ echo $UPDATE_NEEDED
'0'
Why is it $UPDATE_NEEDED is 0 on the 3rd command, but '0' on the 5th command?
What would I need to do to get it to simply set 0? Using UPDATE_NEEDED=0 instead is not an option, as some of the other variables may contain a space (And I'd like to optimistically quote them to have it properly parse spaces).
Also, this is a bit of a XY problem. If anyone knows an easier way to export multiple variables from an executable that can be used later on in the bash script, that could also be useful.
To expand on the answer by Glenn:
When you write something like export UPDATE_NEEDED='0' in Bash code, this is 100% identical to export UPDATE_NEEDED=0. The quotes are used by Bash to parse the command expression, but they are then discarded immediately. Their only purpose is to prevent word splitting and to avoid having to escape special characters. In the same vein, the code fragment 'foo bar' is exactly identical to foo\ bar as far as Bash is concerned: both lead to space being treated as literal rather than as a word splitter.
Conversely, parameter expansion and command substitution follows different rules, and preserves literal quotes.
When you use eval, the command line arguments passed to eval are treated as if they were Bash code, and thus follow the same rules of expansion as regular Bash code, which leads to the same result as (1).
Apparently that Updater project is doing the equivalent of
echo "UPDATE_NEEDED=\'0\' MD5_SUM=\"7e3ad68397421276a205ac5810063e0a\""
It's explicitly outputting the quotes.
When you do export UPDATE_NEEDED='0' MD5_SUM="7e3ad68397421276a205ac5810063e0a",
bash will eventually remove the quotes before actually setting the variables.
I agree with #pynexj, eval is warranted here, although additional quoting is recommended:
eval export "$(dotnet ...)"
I recently converted a shell script from bash to zsh and got a strange error. I had a command like
HOST="User#1.1.1.1"
scp "$BASE_DIR/path/to/file" $HOST:some\\path
This worked fine in bash, but zsh failed with a bad substitution. I fixed this by change $HOST to ${HOST}, but I'm curious as to why this was necessary. Also, strangely, I had a few such scp commands, and all of them "worked" except the first one. However, I ended up with a file called User#1.1.1.1 on my filesystem which was really unexpected. Why did this subtle change make such a big difference?
Two possible problems (1) Extra '$' at the beginning of the assignment, and (2) embedded spaces.
The first potential problem is the assignment in the style $var=foo. In zsh like in other sh-like engines (ksh, bash, ...), the assignment operation is VAR=value - no $.
The second potential problem are the spaces. No spaces are allowed between the variables name, the '=' and the value. Spaces in the value must be escaped (with quotes, or backslash)
Potential correction:
HOST=User#1.1.1.1
scp "$BASE_DIR/path/to/file" $HOST:some\\path
As chepner mentioned in the commments, zsh has modifiers that are added via :. So $HOST:some was interpreted as $HOST:s by zsh.
A list of modifiers can be found here: https://web.cs.elte.hu/local/texinfo/zsh/zsh_23.html
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
Bash is not expanding the ~ character in the argument --home_dir=~. For example:
$ echo --home_dir=~
--home_dir=~
Bash does expand ~ when I leave out the hyphens:
$ echo home_dir=~
home_dir=/home/reedwm
Why does Bash have this behavior? This is irritating, as paths with ~ are not expanded when I specify that path as an argument to a command.
bash is somewhat mistakenly treating home_dir=~ as an assignment. As such, the ~ is eligible for expansion:
Each variable assignment is checked for unquoted tilde-prefixes immediately following a : or the first =. In these cases, tilde expansion is
also performed.
Since --home_dir is not a valid identifier, that string is not mistaken for an assignment.
Arguably, you have uncovered a bug in bash. (I say arguably, because if you use set -k, then home_dir=~ is an assignment, even though it is after, not before, the command name.)
However, when in doubt, quote a string that is meant to be treated literally whether or not it is subject to any sort of shell processing.
echo '--home_dir=~'
Update: This is intentional, according to the maintainer, to allow assignment-like argument for commands like make to take advantage of tilde-expansion. (And commands like export, which for some reason I was thinking were special because they are builtins, but tilde expansion would have to occur before the actual command is necessarily known.)
Like chepner says in their answer, according to the documentation, it shouldn't expand it even in echo home_dir=~. But for some reason it does expand it in any word that even looks like an assignment, and has done so at least as far back as in 3.2.
Most other shells also don't expand the tilde except in cases where it really is at the start of the word, so depending on it working might not be such a good idea.
Use "$HOME" instead if you want it to expand, and "~" if you want a literal tilde. E.g.
$ echo "~" --foo="$HOME"
~ --foo=/home/itvirta
(The more complex cases are harder to do manually, but most of the time it's the running user's own home directory one wants.)
Well, that's because in echo --home_dir=~, the '~' does not begin the word and the output of echo is not considered a variable assignment. Specifically, man bash "Tilde Expansion" provides expansion if
If a word begins with an unquoted tilde character (~); or
variable assignment is checked for unquoted tilde-prefixes immediately following a : or the first =.
You case doesn't qualify as either.
Autoconf scripts have trouble with a filename or pathname with spaces. For example,
./configure CPPFLAGS="-I\"/path with space\""
results in (config.log):
configure:3012: gcc -I"/path with space" conftest.c >&5
gcc: with: No such file or directory
gcc: space": No such file or directory
The compile command from ./configure is ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5' and I am not able to modify this (I could perhaps, but working around autoconf in this way is not a general solution).
I think it comes down to getting a shell variable that contains spaces to be parsed as a single command line variable rather than split at spaces. The simplest shell example I can come up with is to create a file with spaces and attempt to list is with ls with a shell variable as the argument to ls:
$ touch "a b"
$ file="a b"
$ ls $file
ls: a: No such file or directory
ls: b: No such file or directory
This works, but is illegal since in autoconf I can't modify the shell code:
$ ls "$file"
a b
None of the following attempts at quoting things work:
$ file="\"a \"b"; ls $file
ls: "a: No such file or directory
ls: b": No such file or directory
$ file="a\ b"
$ file="a\\ b"
$ file="`echo \\"a b\\"`"
and so on.
Is this impossible to accomplish in shell scripts? Is there a magical quoting that will expand a shell variable with spaces into a single command line argument?
You should try to set the $IFS environment variable.
from man bash(1):
IFS - The Internal Field Separator that is used for word splitting
after expansion and to split lines into words with the read builtin
command. The default value is ''space tab newline''.
For example
IFS=<C-v C-m> # newline
file="a b"
touch $file
ls $file
Don't forget to set $IFS back or strange things will happen.
if you give command
gcc -I"x y z"
in a shell then certainly the single command line parameter "-Ix y z" will be passed to gcc. There is no question to that. That's the whole meaning of double quotes: things inside double quotes are NOT subject to field splitting, and so not subject to $IFS either, for instance.
But you need to be careful about the number of quotes you need. For instance, if you say
file="a b" # 1
and then you say
ls $file # 2
what happens is that the file variable's contents are 'a b', not '"a b"', because the double quotes were "eaten" when line 1 was parsed. The replaced value is then field-separated and you get ls on two files 'a' and 'b'. The correct way to get what you want is
file="a b"; ls "$file"
Now the problem in your original case is that when you set a variable to a string that CONTAINS double quotes, the double quotes are later not interpreted as shell quote symbols but just as normal letters. Which is why when you do something like
file="\"a b\""; ls $file
actually the shell tokenizes the contents of the file variable into '"a' and 'b"' when the ls command is analyzed; the double quote is no longer a shell quote character but just part of the variable's contents. It's analogous to that if you set
file="\$HOME"; ls $file
you get an error that '$HOME' directory does not exist---no environment variable lookup takes place.
So your best options are
Hack autoconf
Do not use path names with spaces (best solution)
Using space in directory names in the Unix world is simply asking for trouble. It's not just the problem of quoting in shell scripts (which needs to be done right anyway): some tools simply cannot cope with spaces in filenames. For instance, you can't (portably) write a Makefile rule that says build baz.o from foo bar/baz.c.
In the case of CPPFLAGS above, I would try one of the following (in order of preference):
Fix the system not use use any space in directory names.
Write a small wrapper around the compiler and call ./configure CC=mygcc. In that case mygcc might be:
#!/bin/sh
gcc "-I/foo bar/include" "$#"
Create a symbolic link (e.g., /tmp/mypath) to the dreaded path and use CPPFLAGS=-I/tmp/mypath.
You want to quote the entire argument, in either of these ways:
./configure "CPPFLAGS=-I/path with space"
./configure CPPFLAGS="-I/path with space"
The ./configure command then sees a single argument
"CPPFLAGS=-I/path with space"
which is parsed as a parameter named«CPPFLAGS» having the value«-I/path with space» (brackets added for clarity).
Using quotes is interesting. From (lightly) reading the bash man page I thought you had to escape the space with \, thus "/path with space" becomes /path\ with\ space I've never tried the quotes, but it seems that it doesn't work generally (your ls example). Escaping works with ls without quoting and without changing IFS.
What happens if you use the "escaping spaces" format of the command?
$ file="\"a b\""
$ eval ls $file
Everything depends on how the variable is used. First, note that if you are using Autoconf, this probably means that make will be used eventually, so that the rules are dictated by make, and in particular, the default make rules. Even though you may want to use your own rules exclusively, things must remain consistent between tools, and some variables have standard meanings, so that you do not want to deviate from them. This is not the case of CPPFLAGS, but this should remain similar to CFLAGS, which is standard. See the POSIX make utility, where variables are simply expanded with standard sh word splitting, which does not provide any quoting mechanism (the field separator is controlled by $IFS, but do not change the IFS variable to accept spaces as normal characters since this will break other things, like being able to provide several -I and/or -L options in such variables with the standard way).
Since there is such a limitation with make, I suppose that it would be useless to try to avoid this limitation in Autoconf.
Now, since a space is necessarily a field separator, the only possibility is to provide pathnames without space characters. If spaces in pathnames were to be supported in the future, this would probably be done via pathname encoding, with decoding at the high-level UI (a bit like with URL's). Alternatively, if you have the choice and really want to use spaces in pathnames, you may use some non-ASCII space (BTW, this is how RISC OS supports space in pathnames, by forcing it to be the no-break space).