This question already has answers here:
How to escape single quotes within single quoted strings
(25 answers)
Closed 4 years ago.
I want to execute the following command by using sh -c (because I want to append similar commands later):
impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://.../foo.parquet' STORED AS PARQUET"
This command works fine. But if I try
sh -c 'impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://../foo.parquet' STORED AS PARQUET"'
I get a syntax error:
LIKE PARQUET hdfs://.../...
^
There must be something wrong with the single quotes from the hdfs path but I can't figure it out. The hdfs path needs to be in quotes. I also tried to backslash them with /' which should actually work according to the docs. I hope someone can help me with that.
You can use Python to tell you how to quote a shell command. Yes, really. :)
python -c '
import sys
try:
from pipes import quote
except ImportError:
from shlex import quote
print(quote(sys.stdin.read().rstrip("\n")))
' <<'EOF'
impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://.../foo.parquet' STORED AS PARQUET"
EOF
...emits as output:
'impala-shell -q "CREATE TABLE test LIKE PARQUET '"'"'hdfs://.../foo.parquet'"'"' STORED AS PARQUET"'
...and indeed, you can succesfully run:
sh -c 'impala-shell -q "CREATE TABLE test LIKE PARQUET '"'"'hdfs://.../foo.parquet'"'"' STORED AS PARQUET"'
Because everything in single-quotes is literal, including backslashes, you need to change to a different quoting context in order to include a single quote literal in a single-quoted string.
That's what '"'"' does: First, it ends the single-quoted context; then it enters a double-quoted context; then it inserts a literal '; then it ends the double-quoted context; then it goes back into a single-quoted context.
Related
I have a py script which gets as an argument data stream char > and >>, for example
python script.py -o 'ls -la /tmp > /tmp/test'
When I'm executing this kind of command the > char perceived with my terminal. Could you help me, how can I pass the > as a symbol so that my console doesn't think of it as a command?
Single quotes already work exactly as you would wish. Probably you are doing something wrong with the string inside of your Python script.
In some more detail, the arguments when Bash is done parsing this command are (one per line)
python
script.py
-o
ls -la /tmp > /tmp/test
(The quotes were useful while Bash was parsing this command line, but they are gone now.)
You should easily be able to verify writhin Python that sys.argv[0] is script.py, sys.argv[1] is -o, and sys.argv[2] is the single-quoted string, sans quotes; all of these are strings (the shell really doesn't have any other primitive data type).
Backslash escaping all shell metacharacters individually would work as well here;
python script.py -o ls\ -la\ /tmp\ \>\ test
or double quoting the string; albeit in the shell, double quotes have slightly different semantics (variables will be interpolated and backticks evaluated, and a backslash can be used to escape backslashes, double quotes, dollar signs, and backticks, unlike in single quotes, where every character is preserved verbatim, and there is no way to escape a character.)
You don't reveal how exactly your code doesn't work, but I'd speculate that your problem is actually the opposite; you end up running something like ls '>' and get an error message that this file does not exist.
For the record, > is a redirection operator, not a command. The shell parses this into
ls
-la
/tmp
with standard oqtput redirected to the file /tmp/test. In terms of Python code,
with open('/tmp/test', 'w') as redirect:
subprocess.run(['ls', '-la', '/tmp'], stdout=redirect)
Of course, if you want to support arbitrary shell features, you need a shell. The simplest fix here is probably
subprocess.run(sys.argv[2], shell=True)
See Actual meaning of shell=True in subprocess but here, I don't see any easy way to avoid the shell, short of by writing your own reimplementation.
In general, when you want to use special characters literally, i.e., without their special meaning, you need to surround a string of characters with single quotation marks(or quotes) to strip all characters within the quotes of any special meaning they might have. It seems that you’re already applied quoting whilst passing your command line argument to the python script. So it should work as expected. For example let’s assume that your python script looks like this:
import argparse
parser = argparse.ArgumentParser(description='Your App')
parser.add_argument('-o', action="store", dest="command")
options = parser.parse_args()
print(options.command)
Invoking this script with python ./script.py -o 'ls -la /tmp > /tmp/test' will produce the below results as expected.
ls -la /tmp > /tmp/test
But you've already mentioned that it is not the behavior you're seeing. So the next thing you might check is the code in your actual python script to determine what’s going on there with the input value. It is most likely that the problem is in your python script. So, the next step I would do is inspecting your python script to understand that magic.
Try by escaping the character using a backslash \ before the greather than > symbol.
I'm currently migrating AIX to Linux. The Oracle script contains a $ in the column name. While fetching through the shell script, I set the escape character to $, but it does not work. The query is like below:
Set escape $
Select c.logoff$time from temp c;
When run from shell script I'm getting "c.logoff invalid identifier".
How can I fix this?
This isn't an SQL*Plus (I assume that's your client) problem, so the set escape isn't doing anything - that's for escaping things SQL*Plus tries to interpret - see the docs.
This is a shell issue/feature. the $time part is being treated as a shell variable, and that doesn't exist, so the final table name doesn't have it. You can escape that at shell level, referring to \$time; e.g. if you're using a heredoc:
sqlplus -s -l usr/pass#db <<EOF
select c.logoff\$time from temp c;
EOF
I get the error executing bash syntax error near unexpected token `('
I know the error is caused by the ')' but I thought placing the commands in-between ' ' is suppose to allow the parenthesis in a directory name. How can I fix this without renaming the name?
The matlab / octave code is:
syscmd=strcat({'bash -c '},{''''},{'cd '},dirpathpls,newdirname,{' && exec bash xfade.sh'},{''''}) %used to run script to join files in stretch directory
system(syscmd);
and it produces what is below:
bash -c 'cd /tmp/h1/clients/04212015142432811_Fs_1000_ahh/pls/03sox_a_Fs_1000_ahh_(000_bit)_(0.0000
0sig_in_deg)_to_(508_bit)_(30.00000sig_in_deg) && exec bash xfade.sh'
please note:
It's being called from inside octave 3.8.1 a math program like matlab
Using ' within a bash command line does allow the use of reserved characters like ( without escaping; however, that is not what you are doing. Everything within your 's is being passed to bash for interpretation, bash isn't interpreting the 's as part of the command. Something like this should work:
syscmd=strcat({'bash -c '},{''''},{'cd "'},dirpathpls,newdirname,{'" && exec bash xfade.sh'},{''''}) %used to run script to join files in stretch directory
system(syscmd);
I don't know matlab/octave, but I hope that conveys the idea. The " should effectively escape the parens. The only pitfall there is if your directory name might have a $ or " in it, in that case, or you have ' AND " in your dir name, things are going to get silly.
As I told you in your other question on this topic: Don't use bash -c; it's not necessary for octave to run an external command, and you're doing nothing but making your life harder by trying.
command=strcat({'cd '''},
strrep(strcat(dirpathpls,newdirname),
'''',
'''"''"'''''),
{''' && exec bash xfade.sh'})
system(syscmd);
Two key differences:
We're using the sh -c implicitly created by the system() call
We're escaping the filenames, preventing any malicious content within them from escaping the quotes and being executed.
How that escaping works:
Single-quoted strings in POSIX shells are ended only by a following single-quote. To insert a literal single quote into them, one needs to end the single-quoted string and then enter a different quoting type. Thus:
'"'"'
...in which the first ' ends the prior quoting type; the " enters a double-quoted context (in which a single-quote literal can be recognized; the ' after it is then your literal single-quote character; the " following ends the double-quoted context, and the final ' resumes a single-quoted context.
This is all made more complicated by doubling the 's to ''s for Octave's syntax; this is how one gets
strrep(content, '''', '''"''"''''')
...to replace all 's with '"'"'s.
Here's a bash script line I'm trying to do:
psql -c 'GRANT ALL PRIVILEGES ON DATABASE "$PROJECT_ID" to "$PROJECT_ID";'
As expected, it doesn't work because bash doesn't interpret anything inside single quotes. Swapping the single and double quotes doesn't work for the same reason.
Solution?
EDIT: What I said about swapping isn't correct -- a bad assumption on my part; however, don't want to swap due to ensure the psql statement still works.
If you're sure that double-quotes are correct in this SQL statement:
psql -c 'GRANT ALL PRIVILEGES ON DATABASE "'"$PROJECT_ID"'" to "'"$PROJECT_ID"'";'
Here's why that works: String quoting shell is done on a character-by-character basis; we don't need to quote the whole string the same way, and so can concatenate several differently-quoted string subsets together.
In this case, those substrings are:
'GRANT ALL PRIVILEGES ON DATABASE "'
"$PROJECT_ID"
'" to "'
"$PROJECT_ID"
'";'
Thus, when we want to pass a literal " as part of our string, we enclose it in single-quotes; and when we want to expand a variable, we put it in double quotes.
I have the following which I'm trying to script. I've hit a wall with trying to cater for the mixed quotes as well as asterix.
This is what I have
cmd_1="/usr/local/nz/bin/nzsql -d ${SOURCE_DB} -c "
cmd_2="\"create external table '${EXTERNAL_PATH}${EXTERNAL_FILE}'"
cmd_3=" USING (QUOTEDVALUE 'DOUBLE' ESCAPECHAR '\' DELIM ',' TIMESTYLE '24HOUR' LOGDIR '${EXTERNAL_PATH}' ENCODING 'INTERNAL')"
cmd_4=" as select * from ${SOURCE_TABLE}\""
cmd=${cmd_1}${cmd_2}${cmd_3}${cmd_4}
Which when echo'd looks fine and in fact runs with a copy and paste to the command line.
/usr/local/nz/bin/nzsql -d mydatDba -c "create external table '/datawarehouse/development/externaldata/output/EXT_mytable.csv' USING (QUOTEDVALUE 'DOUBLE' ESCAPECHAR '\' DELIM ',' TIMESTYLE '24HOUR' LOGDIR '/datawarehouse/development/externaldata/output/' ENCODING 'INTERNAL') as select * from mytable"
To get it to echo correctly, I did have to wrap ${cmd} in double quotes and so did the same for the execute which looks like this.
"${cmd}"
When running from the shell script however I receive : No such file or directory. I have verified the directories involved and as I mentioned can actually copy and paste what's generated to the command line and it works fine.
I can only assume that what is being echoed in both my echo command and the error is not actually what's being executed.
Advise on how to get the 'real' command echoed and or on my use of quotes would be greatly appreciated.
Thanks.
Use the built-in eval command:
eval "$cmd"