I'm trying to navigate down to a subfolder in a bash shell. The name of the subfolder is:
Let's Go Play!
I cannot figure out how to escape the single quote (apostrophe) or the exclamation point.
I have tried
cd "Let's Go Play\!"
cd "Let\'s Go Play\!"
Thanks.
The correct form is
cd "Let's Go Play!"
Inside double quotes, backslashes are not special unless they come before a newline, a quote, a backslash, a dollar sign or a backtick. Backslash-newline is removed altogether; a backslash followed by one of the other four characters in that list is removed and the character loses its special significance.
Inside single quotes, backslashes are never removed and have no special significance. Consequently, it is impossible to insert a single quote into a single-quoted string and so there is no single-quoted form of the above cd command. However, you can concatenate words, so you could write:
cd 'Let'"'"'s Go Play!'
Outside of quoted words, backslashes are more general. A backslash followed by any character other than a newline character is removed from the input and the following character becomes an ordinary character (even if it were ordinary already). Backslash-newline is removed entirely from the input, so that there is no way to insert a newline character into an unquoted string.
So you could have written:
cd Let\'s\ Go\ Play\!
But the double-quoted version one seems simpler.
Exclamations points are an extension to the Posix standard (the above rules comes directly from the Posix standard), and the bash implementation is a bit quirky and sometimes really annoying. Exclamation points introduce history expansion, unless they are inside single quotes, are preceded by a backslash, or are followed by whitespace or either an equals sign or (if shell option extglob is enabled) an open parenthesis. Inside double quotes, an exclamation point is also not special just before the closing quote. (You can change the history expansion character to something other than an exclamation point so technically I should write "the history expansion character".)
Even though a backslash makes an exclamation mark unspecial, the backslash is not removed from the input stream unless it would have been removed by the Posix rules. So the exclamation point in
echo "a\!b"
is an ordinary character (it is preceded by a backslash), but the backslash is also an ordinary character (it is not followed by one of the characters in the double-quote list), so the result is
a\!b
(Although I copied those rules from the bash manual, I know there are some other cases where history expansion is suppressed, such as when the exclamation point is part of a parameter expansion such as $! or ${!name}. And I think there are more of these exceptions that I can't remember off-hand.)
I find all that so annoying, and I rely so little on history expansion, that I simply turn it off by adding set +H to my bash startup file ~/.bashrc. If you turn history expansion off, then exclamation points lose all special significance. However, there are people who seem to really like history expansion, and if you're one of them, more power to you.
Related
When using BASH Parameter Expansion, string that variable expands into can be quoted/escaped, which works fine, except when the single quotes are used and the whole variable is escaped in double quotes:
$ echo "${var:-\\a}"
\a # ok
$ echo "${var:-"\\a"}"
\a # ok
$ echo "${var:-$'\\a'}"
\a # ok
$ echo "${var:-'\a'}"
'\a' # wtf?
Interestingly, $' ' quotes work normally, while ' ' don't. Single quotes start working correctly if the variable itself is not quoted:
$ echo ${var:-'\a'}
\a
But, that can lead to other issues if $var itself contains whitespace characters.
Is there any good reason for this inconsistency?
I think this is the most relevant quote from the source code (y.tab.c):
/* Based on which dolstate is currently in (param, op, or word),
decide what the op is. We're really only concerned if it's % or
#, so we can turn on a flag that says whether or not we should
treat single quotes as special when inside a double-quoted
${...}. This logic must agree with subst.c:extract_dollar_brace_string
since they share the same defines. */
/* FLAG POSIX INTERP 221 */
[...]
/* The big hammer. Single quotes aren't special in double quotes. The
problem is that Posix used to say the single quotes are semi-special:
within a double-quoted ${...} construct "an even number of
unescaped double-quotes or single-quotes, if any, shall occur." */
/* This was changed in Austin Group Interp 221 */
It's not exactly clear to me why single quotes aren't special, but it seems like a conscious choice made after long (and I've been told contentious) debate preceding the change. But the fact is (if I am summarizing this correctly), single quotes here are just regular characters, not syntactic quotes, and are treated literally.
I'm fairly new to Bash and I'm having trouble working out what is happening to my input as it is interpreted. Specifically, when escaping occurs relative to the other expansion steps.
From what I've read, bash does the following (in order):
brace expansion
tilde expansion
parameter and variable expansion
command substitution
arithmetic expansion
word splitting
filename expansion
But this list doesn't include when it converts all escape sequences e.g. '\\' into their meanings e.g. '\'. That is, if I want to print a backslash character. The command to run is
echo \\
not
echo \
So the syntax required for the semantics of a backslash character is two backslashes. This must be converted into a single slash representation internally.
It seems to be sometime before command substitution as I found out with a small test program.
So, my question is: When does this step take place? (or a complete list of the bash interpretation loop would be perfect)
and also, are there any other subtleties in the interpreter that are likely to catch me out? (related to knowing the complete list i guess)
From the man page's Expansion section, just before the Redirection section.
Quote Removal
After the preceding expansions, all unquoted occurrences of the characters \, ', and " that did not result from one of the above expansions
are removed.
Quote removal is one final process after the seven expansions you list.
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
Hi I have this little command to copy files in a batch, which will help because I do this specific copy multiple times a day. The problem occurs while using the xcopy command. Everything is in order, but I am receiving this error: "Invalid path 0 files copied". Here is the code:
C:\Windows\System32\xcopy /Y "C:\Users\Ryan\Desktop\mmars_pub\" "C:\Users\Ryan\Desktop\Dropbox\MMARS\mmars_pub\"
I'm using the full path to the xcopy executable because I was having trouble configuring the path environment variable to function properly. I would imagine that it shouldn't affect the result though. I read somewhere about the "Prevent MS-DOS-based programs from detecting Windows" checkbox that should fix the issue, but I just can't seem to find that. Any help appreciated.
Original answer
Remove the ending backslash from the source folder path
C:\Windows\System32\xcopy.exe /Y "C:\Users\Ryan\Desktop\mmars_pub" "C:\Users\Ryan\Desktop\Dropbox\MMARS\mmars_pub\"
edited 2015/10/01
While the original question used a literal path, and the indicated solution will solve the problem, there is another option. For literal paths and in cases where the path is inside a variable and could (or not) end in a backslash, it is enough to ensure that the ending backslash (if present) is separated from the quote, including an ending dot.
xcopy /y "x:\source\." "x:\target"
xcopy /y "%myVariable%." "x:\target"
This ending dot will not interfere in files/folders names. If there is and ending backslash, the additional dot will simply refer to the same folder. If there is not ending backslash as in windows files and folders can not end their names with a dot, it will be discarded.
BUT if the output of the xcopy command will be processed, remember that this additional dot will be included in the paths shown.
note: The solutions are above the line. Keep reading if interested on why/where there is a problem.
Why xcopy "c:\source\" "d:\target\" fails but xcopy "c:\source" "d:\target\" works?
Both commands seems to have valid path references, and ... YES! both are valid path references, but there are two elements that work together to make the command fail:
the folder reference is quoted (note: it should be quoted, it is a good habit to quote paths as you never know when they will contain spaces or special characters)
xcopy is not an internal command handled by cmd but an executable file
As xcopy is an external command, its arguments are not handled following the cmd parser command line logic. They are handled by the Microsoft C startup code.
This parser follows two sets of rules, official rules
Arguments are delimited by white space, which is either a space or a tab.
A string surrounded by double quotation marks is interpreted as a single argument, regardless of white space contained within. A quoted
string can be embedded in an argument. Note that the caret (^) is not
recognized as an escape character or delimiter.
A double quotation mark preceded by a backslash, \", is interpreted as a literal double quotation mark (").
Backslashes are interpreted literally, unless they immediately precede a double quotation mark.
If an even number of backslashes is followed by a double quotation mark, then one backslash (\) is placed in the argv array for every
pair of backslashes (\\), and the double quotation mark (") is
interpreted as a string delimiter.
If an odd number of backslashes is followed by a double quotation mark, then one backslash (\) is placed in the argv array for every
pair of backslashes (\\) and the double quotation mark is interpreted
as an escape sequence by the remaining backslash, causing a literal
double quotation mark (") to be placed in argv.
and undocumented/non official rules (How Command Line Parameters Are Parsed)
Outside a double quoted block a " starts a double quoted block.
Inside a double quoted block a " followed by a different character (not another ") ends the double quoted block.
Inside a double quoted block a " followed immediately by another " (i.e. "") causes a single " to be added to the output, and the
double quoted block continues.
This parser sees the sequence \" found at the end of the "first" argument as a escaped quote that does not end/closes the argument, it is seen as part or the argument. And the "starting" quote of the "second" argument is just ending the double quoted block BUT not ending the argument, remember that arguments are delimited by white space.
So while it seems that the command line arguments are
v v v......argument delimiters
v.........v v..........v ......quoted blocks
xcopy "x:\souce\" "x:\target\"
^.......^ ^........^ ......argument data
arg #1 arg #2
arg #1 = x:\source\
arg #2 = x:\target\
the actual argument handled by xcopy is
v v .....argument delimiters
v......................v .....quoted block
xcopy "x:\souce\" "x:\target\"
^.....................^ .....argument data
arg #1
arg #1 = x:\source" x:\target"
When the ending backslash is removed or the additional dot included, the closing quote in the argument will not be escaped, it will close the quoted block and the space between arguments will be seen as a delimiter.
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'