I was surprised that the following is a valid Parameter Expansion. Notice there are unescaped double quotes within double quotes:
result="${var1#"$var2"}"
Can someone please parse this for me?
There are double quotes nested in curly brackets which is OK.
But none of them is needed in this case.
result=${var1#$var2}
works the same even for values containing spaces and newlines.
The answer is that they get parsed separately. Let's take a simplified tour of the string.
result="${var1#"$var2"}" doesn't actually need any quotes in this case, but look over the string anyway...
result="...
The Parser says meh, it's an assignment, I know what to do with this, I'll ignore these, they aren't hurting anything, but now I have to find the terminating match. Then it reads the value after the quote, byte by byte, looking for the terminating double-quote. This starts a new context-1.
result="${...
Once it sees the open curly, it knows that the terminating quote cannot happen until it sees the matching closing curly. It starts a new context-2.
result="${var1#"...
Seeing a new double quote in this subcontext make this one the opening quote of an internal new context-3.
result="${var1#"$var2"...
When it sees this double-quote it matches it to the previous one, closing context-3, dropping back into context-2.
result="${var1#"$var2"}...
This close-curly allows it to close the still-open context-2, dropping back into context-1.
result="${var1#"$var2"}"
And finding this now-closing double-quote allows it to close context-1. The following newline may be used as a terminating character for the entire term, so it can be evaluated and assigned.
Backslash-Quoting the internal double-quotes, for example, would have added them to the string-term used for the tail trim, which would likely have failed because of it.
$: var1=aaa
$: var2=a
$: result="${var1#"$var2"}"
$: echo $result # does what you want/expect
aa
$: result="${var1#\"$var2\"}" # does NOT
$: echo $result
aaa
Doing it without the quotes, the parser knows this is an assignment and handles the values a little differently as mentioned in comments, but generally kinda treating them as if they were quoted.
$: result=${var1#$var2}
$: echo $result
aa
This means it doesn't have to deal with context-1 or context-3, and only has the curlies to worry about. The end result is the same.
Better?
Related
Essentially what I am trying to do is take a string with a bunch of text and if it has a substring of "$$" to replace it with a substring of "$$$"
ex:
string="abcde\$\$fghi"
# Modify string
echo $string
# ^ should give "abcde$$$fghi"
I have been at this for like 2 hours now and it seems like a very simple thing, so if anyone could provide some help then I would greatly appreciate it. Thanks!
EDIT: Changed original string in the question from "abcde$$fghi" to "abcde\$\$fghi"
$$ is a special variable in the shell, it contains the ID of the current process. The variables are expanded in double quotes, therefore string does not contain $$ but a number (the PID of shell) instead.
Enclose the string in apostrophes (single quotes) to get $$ inside it.
The replacement you need can be done in multiple ways. The simplest way (probably) and also the fastest way (for sure) is to use / in the parameter expansion of $string:
echo "${string/'$$'/'$$$'}"
To make it work you have to use the same trick as before: wrap $$ and $$$ in single quotes to prevent the shell replace them with something else. The quotes around the entire expression are needed to preserve the space characters contained by $string, otherwise the line is split to words by whitspaces and and echo outputs these words separated by one space character.
Check it online.
If you quote the string with single quote marks (i.e. string='abcde$$fghi') you can do the replacement with echo "${string/'$$'/'$$$'}"
Edit: this is basically what #axiac said in their comment
When using BASH Parameter Expansion, string that variable expands into can be quoted/escaped, which works fine, except when the single quotes are used and the whole variable is escaped in double quotes:
$ echo "${var:-\\a}"
\a # ok
$ echo "${var:-"\\a"}"
\a # ok
$ echo "${var:-$'\\a'}"
\a # ok
$ echo "${var:-'\a'}"
'\a' # wtf?
Interestingly, $' ' quotes work normally, while ' ' don't. Single quotes start working correctly if the variable itself is not quoted:
$ echo ${var:-'\a'}
\a
But, that can lead to other issues if $var itself contains whitespace characters.
Is there any good reason for this inconsistency?
I think this is the most relevant quote from the source code (y.tab.c):
/* Based on which dolstate is currently in (param, op, or word),
decide what the op is. We're really only concerned if it's % or
#, so we can turn on a flag that says whether or not we should
treat single quotes as special when inside a double-quoted
${...}. This logic must agree with subst.c:extract_dollar_brace_string
since they share the same defines. */
/* FLAG POSIX INTERP 221 */
[...]
/* The big hammer. Single quotes aren't special in double quotes. The
problem is that Posix used to say the single quotes are semi-special:
within a double-quoted ${...} construct "an even number of
unescaped double-quotes or single-quotes, if any, shall occur." */
/* This was changed in Austin Group Interp 221 */
It's not exactly clear to me why single quotes aren't special, but it seems like a conscious choice made after long (and I've been told contentious) debate preceding the change. But the fact is (if I am summarizing this correctly), single quotes here are just regular characters, not syntactic quotes, and are treated literally.
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
A="echo 'q'"
$A
I got the result 'q'
but if I type echo 'q' directly, it's q (without single quote)
So, I wonder what rule does bash follow while facing single quotes inside the double quotes.
The original problem is
A="curl http://123.196.125.62/send -H 'Host: poj.org' -e http://poj.org/send"
$A
I got curl: (6) Couldn't resolve host 'poj.org''
it will be all right if I just type the command into the terminal..
P.S. I'd like to use $A for excuting the command inside A
Please see BashFAQ/050: I'm trying to put a command in a variable, but the complex cases always fail!
It's best to avoid putting commands in variables for the reason you've experienced, among others.
You should use a function and pass it arguments.
Why do you want to do this rather than simply executing the command directly?
If you must do it, use an array:
A=(curl http://123.196.125.62/send -H 'Host: poj.org' -e http://poj.org/send)
${A[#]}
Regarding the treatment of single quotes within double quotes, they are treated literally and as part of the rest of the string. Here is the relevant paragraph from man bash:
Enclosing characters in double quotes preserves the literal value of
all characters within the quotes, with the exception of $, `, \, and,
when history expansion is enabled, !. The characters $ and ` retain
their special meaning within double quotes. The backslash retains its
special meaning only when followed by one of the following characters:
$, `, ", \, or <newline>. A double quote may be quoted within double
quotes by preceding it with a backslash. If enabled, history expansion
will be performed unless an ! appearing in double quotes is escaped
using a backslash. The backslash preceding the ! is not removed.
If you want to "save" a command for later execution, you do NOT want a variable. You want a function.
a() { curl http://123.196.125.62/send -H 'Host: poj.org' -e http://poj.org/send; }
Putting code in variables is bad since variables are containers for data not code. Additionally, you are seeing the problem because $A does NOT execute the bash code in A, what it really does is split the value of A into words, then it performs Pathname Expansion on these words, and as a result of those two operations, it executes a program named by the first resulting word and passes the other words as arguments. In your particular case, this is what happens (I use [] to indicate "units"):
A: [echo 'q']
after wordsplitting: [echo] ['q']
after pathname expansion: [echo] ['q']
Now bash looks for a program called echo and passes the argument 'q' to it.
This is NOT executing bash code, because if you execute echo 'q' as bash code, bash removes the single quotes after it's done with them. Similarly, you cannot do pipes, redirection et al. like this, because they too are bash syntax (just like your single quotes).
Recap: never put code in bash variables. never leave parameters unquoted (if you think doing that fixes something, you are wrong, you've just made things worse, go fix the real problem). The solution is to use a function.
You'd better use backquotes in this case:
A=`curl http://123.196.125.62/send -H 'Host: poj.org' -e 'http://poj.org/send'`