Here's a bash script line I'm trying to do:
psql -c 'GRANT ALL PRIVILEGES ON DATABASE "$PROJECT_ID" to "$PROJECT_ID";'
As expected, it doesn't work because bash doesn't interpret anything inside single quotes. Swapping the single and double quotes doesn't work for the same reason.
Solution?
EDIT: What I said about swapping isn't correct -- a bad assumption on my part; however, don't want to swap due to ensure the psql statement still works.
If you're sure that double-quotes are correct in this SQL statement:
psql -c 'GRANT ALL PRIVILEGES ON DATABASE "'"$PROJECT_ID"'" to "'"$PROJECT_ID"'";'
Here's why that works: String quoting shell is done on a character-by-character basis; we don't need to quote the whole string the same way, and so can concatenate several differently-quoted string subsets together.
In this case, those substrings are:
'GRANT ALL PRIVILEGES ON DATABASE "'
"$PROJECT_ID"
'" to "'
"$PROJECT_ID"
'";'
Thus, when we want to pass a literal " as part of our string, we enclose it in single-quotes; and when we want to expand a variable, we put it in double quotes.
Related
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
This question already has answers here:
How to escape single quotes within single quoted strings
(25 answers)
Closed 4 years ago.
I want to execute the following command by using sh -c (because I want to append similar commands later):
impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://.../foo.parquet' STORED AS PARQUET"
This command works fine. But if I try
sh -c 'impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://../foo.parquet' STORED AS PARQUET"'
I get a syntax error:
LIKE PARQUET hdfs://.../...
^
There must be something wrong with the single quotes from the hdfs path but I can't figure it out. The hdfs path needs to be in quotes. I also tried to backslash them with /' which should actually work according to the docs. I hope someone can help me with that.
You can use Python to tell you how to quote a shell command. Yes, really. :)
python -c '
import sys
try:
from pipes import quote
except ImportError:
from shlex import quote
print(quote(sys.stdin.read().rstrip("\n")))
' <<'EOF'
impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://.../foo.parquet' STORED AS PARQUET"
EOF
...emits as output:
'impala-shell -q "CREATE TABLE test LIKE PARQUET '"'"'hdfs://.../foo.parquet'"'"' STORED AS PARQUET"'
...and indeed, you can succesfully run:
sh -c 'impala-shell -q "CREATE TABLE test LIKE PARQUET '"'"'hdfs://.../foo.parquet'"'"' STORED AS PARQUET"'
Because everything in single-quotes is literal, including backslashes, you need to change to a different quoting context in order to include a single quote literal in a single-quoted string.
That's what '"'"' does: First, it ends the single-quoted context; then it enters a double-quoted context; then it inserts a literal '; then it ends the double-quoted context; then it goes back into a single-quoted context.
I get the error executing bash syntax error near unexpected token `('
I know the error is caused by the ')' but I thought placing the commands in-between ' ' is suppose to allow the parenthesis in a directory name. How can I fix this without renaming the name?
The matlab / octave code is:
syscmd=strcat({'bash -c '},{''''},{'cd '},dirpathpls,newdirname,{' && exec bash xfade.sh'},{''''}) %used to run script to join files in stretch directory
system(syscmd);
and it produces what is below:
bash -c 'cd /tmp/h1/clients/04212015142432811_Fs_1000_ahh/pls/03sox_a_Fs_1000_ahh_(000_bit)_(0.0000
0sig_in_deg)_to_(508_bit)_(30.00000sig_in_deg) && exec bash xfade.sh'
please note:
It's being called from inside octave 3.8.1 a math program like matlab
Using ' within a bash command line does allow the use of reserved characters like ( without escaping; however, that is not what you are doing. Everything within your 's is being passed to bash for interpretation, bash isn't interpreting the 's as part of the command. Something like this should work:
syscmd=strcat({'bash -c '},{''''},{'cd "'},dirpathpls,newdirname,{'" && exec bash xfade.sh'},{''''}) %used to run script to join files in stretch directory
system(syscmd);
I don't know matlab/octave, but I hope that conveys the idea. The " should effectively escape the parens. The only pitfall there is if your directory name might have a $ or " in it, in that case, or you have ' AND " in your dir name, things are going to get silly.
As I told you in your other question on this topic: Don't use bash -c; it's not necessary for octave to run an external command, and you're doing nothing but making your life harder by trying.
command=strcat({'cd '''},
strrep(strcat(dirpathpls,newdirname),
'''',
'''"''"'''''),
{''' && exec bash xfade.sh'})
system(syscmd);
Two key differences:
We're using the sh -c implicitly created by the system() call
We're escaping the filenames, preventing any malicious content within them from escaping the quotes and being executed.
How that escaping works:
Single-quoted strings in POSIX shells are ended only by a following single-quote. To insert a literal single quote into them, one needs to end the single-quoted string and then enter a different quoting type. Thus:
'"'"'
...in which the first ' ends the prior quoting type; the " enters a double-quoted context (in which a single-quote literal can be recognized; the ' after it is then your literal single-quote character; the " following ends the double-quoted context, and the final ' resumes a single-quoted context.
This is all made more complicated by doubling the 's to ''s for Octave's syntax; this is how one gets
strrep(content, '''', '''"''"''''')
...to replace all 's with '"'"'s.
I have a script which can overwrite values in a configuration file using options, for example, option --password can overwrite the setting in the configuration file (please note, this is not a discussion about security). However a password can contain contain characters, that are by bash, recognized as special characters, and those characters needs to be escaped or placed within " ".
Now I understand this. Although I can't say whom will be using this script in the future, so I would like to save he or she the trouble of having an incorrect password simply because, he or she forgot to place the password within " " or escape the special character.
What is the best way of dealing with such an input?
Thanks in advance.
Hm.. Double quotes are not enough. Must use single quotes, because the rare situation, for example
mycommand --password "AAA$PWD" #is worng for any exported environment varname
mycommand --password 'AAA$PWD' #ok
Here is no way avoid this, because your users using a sort of shell, what have variable expansions and metachar-globbing. Your command getting already expanded args, so here is no way catch this in your script.
The only way - as #Bohemian told above - reading the password from a tty. You can write a simple wrapper around your current script, what will read the password from a tty and after will execute your script with properly escaped --pasword argument.
There is a simple, but not very intuitive solution.
Encapsulate the character that is causing problem with '"CARACTER"'. And put all the password string between single quotes.
In my case the character that was causing the problem was the ' (single quote).
So if I have a command like that:
mycommand --password 'AAA'PWD'
Replace by that:
mycommand --password 'AAA'"'"'PWD'
Now my password is a concatenation of three strings. Explanation:
'AAA' + "'" + 'PWD' # plus sign is just to make clear the contatenation.
That's works.