My question targets Zsh, but from what I tried, it seems to apply to POSIX shell and bash as well:
I want to write a string containing literal $ characters (no interpolation intended) and single quotes. Since the chapter on QUOTING in the Zsh man page says about single-quoted strings:
A literal ' character can be included in the string by using the \' escape.
I tried something like this (in an interactive zsh, before doing it in a script):
echo 'a$b\'c'
I expected that this would print a$b'c, but zsh tells me that I have an unclosed quote.
I am aware that I can use as a workaround
echo 'a$'"b'C"
but I still would like to know, why my original attempt failed.
The problem was my interpretation of the man-page, and I must blame myself that I have left out of my citation of the page one part which I thought is unimportant, but actually is relevant for this case. Here again the full sentence of the man page:
A string enclosed between '$'' and ''' is processed the same way as the
string arguments of the print builtin, and the resulting string is
considered to be entirely quoted. A literal ''' character can be
included in the string by using the '\'' escape.
Since Zsh has two ways to write a single-quoted String (one where the quotation, like in bash, starts with ' and one where it starts with $'), I understood the between ... and part that it meant "in $' and in ' -quoted strings, the \' escape works.
This interpretation is incorrect. What the man page means is that a quoted string which starts with $' and ends with ', can use that escape.
Hence my example can written as
echo $'a$b\'c'
Related
I want to display a string in Bash like this
I'm a student
Of course you can do it like this
echo "I'm a student"
But how to accomplish this while using single quote around the string ?
echo 'I\'m a student'
does not work. But the following works:
echo $'I\'m a student'
From the man page of bash:
A single quote may not occur between single quotes, even when preceded
by a backslash.
....
Words of the form $'string' are treated specially. The word
expands to string, with backslash-escaped characters replaced as
specified by the ANSI C standard.
The "ugly" solution mentioned by Glenn Jackman should actually be listed as a top level answer. It works well and is actually beautiful in some situations.
'I'"'"'m a student'
This ends the single quoted string after I then immediately starts a double quoted string containing a single quote and then starts another single quoted string. Bash then concatenates all contiguous strings into one.
Beautiful!
The example below works because the escaped single quote \' is technically between two single-quoted arguments
echo 'I'\''m a student'
Another way to workaround is to use printf instead echo and escape the required single quote with \x27:
printf 'I\x27m a student!\n'
I propose
echo "I'm a student"
like in other languages.
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
In C#, there is a verbatim string so that,
string c = "hello \t world"; // hello world
string d = #"hello \t world"; // hello \t world
I am new to shell script, is there a similar method in shell?
Because I have many folders with the name like "Apparel & Accessories > Clothing > Activewear", I want to know if there is a easy way to process the escape characters without write so many .
test.sh
director="Apparel & Accessories > Clothing > Activewear"
# any action to escape spaces, &, > ???
hadoop fs -ls $director
For definining the specific string in your example, Apparel & Accessories > Clothing > Activewear, either double quotes or single quotes will work; referring to it later is a different story, however:
In the shell (any POSIX-compatible shell), how you refer to a variable is just as important as how you define it.
To safely refer to a previously defined variable without side-effects, enclose it in double quotes, e.g., "$directory".
To define [a variable as] a literal (verbatim) string:
(By contrast, to define a variable with embedded variable references or embedded command substitutions or embedded arithmetic expressions, use double quotes (").)
If your string contains NO single quotes:
Use a single-quoted string, e.g.:
directory='Apparel & Accessories > Clothing > Activewear'
A single-quoted string is not subject to any interpretation by the shell, so it's generally the safest option for defining a literal. Note that the string may span multiple lines; e.g.:
multiline='line 1
line 2'
If your string DOES contain single quotes (e.g., I'm here.) and you want a solution that works in all POSIX-compatible shells:
Break the string into multiple (single-quoted) parts and splice in single-quote characters:
Note: Sadly, single-quoted strings cannot contain single quotes, not even with escaping.
directory='I'\''m here.'
The string is broken into into single-quoted I, followed by literal ' (escaped as an unquoted string as \'), followed by single-quoted m here.. By virtue of having NO spaces between the parts, the result is a single string containing a literal single quote after I.
Alternative: if you don't mind using a multiline statement, you can use a quoted here document, as described at the bottom.
If your string DOES contain single quotes (e.g., I'm here.) and you want a solution that works in bash, ksh, and zsh:
Use ANSI-C quoting:
directory=$'I\'m here.'
Note: As you can see, ANSI-C quoting allows for escaping single quotes as \', but note the additional implications: other \<char> sequences are subject to interpretation, too; e.g., \n is interpreted as a newline character - see http://www.gnu.org/software/bash/manual/bash.html#ANSI_002dC-Quoting
Tip of the hat to #chepner, who points out that the POSIX-compatible way of directly including a single quote in a string to be used verbatim is to use read -r with a here document using a quoted opening delimiter (the -r option ensures that \ characters in the string are treated as literals).
# *Any* form of quoting, not just single quotes, on the opening EOF will work.
# Note that $HOME will by design NOT be expanded.
# (If you didn't quote the opening EOF, it would.)
read -r directory <<'EOF'
I'm here at $HOME
EOF
Note that here documents create stdin input (which read reads in this case). Therefore, you cannot use this technique to directly pass the resulting string as an argument.
use strong quotes i.e. 'string', allowing escape char or special char for string.
e.g. declare director='Apparel & Accessories > Clothing > Activewear'
also using declare is a good practice while declaring variable.
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
I have a bash script,
echo 'abcd'
in shell, I want to show ab'c'd and I have tried following approach but without success
echo 'ab\'c\'d'
I am asking is it possible to show single quote in single quoted text?
From the bash manual section on Single Quotes:
A single quote may not occur between single quotes, even when preceded by a backslash.
You'll need to use double quotes instead. It's not pretty, but the following gives the output you are looking for:
echo 'ab'"'"'c'"'"'d'
A bash-specific feature, not part of POSIX, is a $'...'-quoted string:
echo $'ab\'c\'d'
Such a string behaves identically to a single-quoted string, but does allow for a selection of \-escaped characters (such as \n, \t, and yes, \').