I have the following which I'm trying to script. I've hit a wall with trying to cater for the mixed quotes as well as asterix.
This is what I have
cmd_1="/usr/local/nz/bin/nzsql -d ${SOURCE_DB} -c "
cmd_2="\"create external table '${EXTERNAL_PATH}${EXTERNAL_FILE}'"
cmd_3=" USING (QUOTEDVALUE 'DOUBLE' ESCAPECHAR '\' DELIM ',' TIMESTYLE '24HOUR' LOGDIR '${EXTERNAL_PATH}' ENCODING 'INTERNAL')"
cmd_4=" as select * from ${SOURCE_TABLE}\""
cmd=${cmd_1}${cmd_2}${cmd_3}${cmd_4}
Which when echo'd looks fine and in fact runs with a copy and paste to the command line.
/usr/local/nz/bin/nzsql -d mydatDba -c "create external table '/datawarehouse/development/externaldata/output/EXT_mytable.csv' USING (QUOTEDVALUE 'DOUBLE' ESCAPECHAR '\' DELIM ',' TIMESTYLE '24HOUR' LOGDIR '/datawarehouse/development/externaldata/output/' ENCODING 'INTERNAL') as select * from mytable"
To get it to echo correctly, I did have to wrap ${cmd} in double quotes and so did the same for the execute which looks like this.
"${cmd}"
When running from the shell script however I receive : No such file or directory. I have verified the directories involved and as I mentioned can actually copy and paste what's generated to the command line and it works fine.
I can only assume that what is being echoed in both my echo command and the error is not actually what's being executed.
Advise on how to get the 'real' command echoed and or on my use of quotes would be greatly appreciated.
Thanks.
Use the built-in eval command:
eval "$cmd"
Related
This question already has answers here:
How to escape single quotes within single quoted strings
(25 answers)
Closed 4 years ago.
I want to execute the following command by using sh -c (because I want to append similar commands later):
impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://.../foo.parquet' STORED AS PARQUET"
This command works fine. But if I try
sh -c 'impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://../foo.parquet' STORED AS PARQUET"'
I get a syntax error:
LIKE PARQUET hdfs://.../...
^
There must be something wrong with the single quotes from the hdfs path but I can't figure it out. The hdfs path needs to be in quotes. I also tried to backslash them with /' which should actually work according to the docs. I hope someone can help me with that.
You can use Python to tell you how to quote a shell command. Yes, really. :)
python -c '
import sys
try:
from pipes import quote
except ImportError:
from shlex import quote
print(quote(sys.stdin.read().rstrip("\n")))
' <<'EOF'
impala-shell -q "CREATE TABLE test LIKE PARQUET 'hdfs://.../foo.parquet' STORED AS PARQUET"
EOF
...emits as output:
'impala-shell -q "CREATE TABLE test LIKE PARQUET '"'"'hdfs://.../foo.parquet'"'"' STORED AS PARQUET"'
...and indeed, you can succesfully run:
sh -c 'impala-shell -q "CREATE TABLE test LIKE PARQUET '"'"'hdfs://.../foo.parquet'"'"' STORED AS PARQUET"'
Because everything in single-quotes is literal, including backslashes, you need to change to a different quoting context in order to include a single quote literal in a single-quoted string.
That's what '"'"' does: First, it ends the single-quoted context; then it enters a double-quoted context; then it inserts a literal '; then it ends the double-quoted context; then it goes back into a single-quoted context.
i have PHP code use some bash codes which the PHP code can run it, and its have a bug to make RCE in bash,
the command would be "$(id)" command executed as well
but if i execute any other command like "ls -la" its have a space
the space replaced automatically with "-"
i checked the source as well and i found the following chars spaces,',`,/,\,<,>,?,&,| are filtered
how to bypass them and execute command like "wget link" and run it perfect
****UPDATE****
the following code i add as a live example.
send the command in sendcmd function
`https://pastebin.com/raw/1MfR6aic`
This is (example) output from id
uid=1000(ibug) gid=1000(ibug)
Since these characters aren't filtered, you can get an unfiltered space like this:
ID=$(id)
echo${ID:14:1}foo
Now you have space. You can get virtually any character with echo -e and then eval an expression.
I tried your PHP code and found this working:
sendcmd("http://52.27.167.139", "{echo,hello}");
Just wrap then in braces and use commas. The shell will expand the brace to
echo hello
I'm writing a shell script with user input like so:
echo -n "Enter string for Read Group parameter: "
read readgroup
I am passing this variable to another application that requires this parameter to be a string that starts with # and is enclosed in quotes on the command line.
This is the correct command-line syntax:
application -R '#text'
application -R "#text"
These commands would produce errors if run on command line:
application -R #text
application -R "text"
application -R 'text'
In my script, I have tried putting escaped quotes in the passed variable:
echo $readgroup
'#text'
application -R "$readgroup" # error: does not start with #
Or passing it without the quotes:
echo $readgroup
#text
application -R "$readgroup" # error: no parameter entry recognized
application -R \""$readgroup"\" # error: does not start with #
application -R "'"'$readgroup'"'" # error: does not start with #
And tried other solutions from stackoverflow like arrays:
readgroup=("#text")
application -R "${readgroup[#]}" # error: does not start with #
The problem is that when I include the quotes in the original variable or escape the quotes on the command call, the application interprets it as a literal string (e.g. beginning with a quote and therefore invalid, since it must begin with an #). However when I don't include the quotes, it returns an error because there are no quotes for the application call.
Is there a way to force the shell to interpret the escaped quotes (in the script) as actual command line quotes (parameters) or to add those quotes explicitly in the application call in the script?
Or is a different approach considered better, such as using arguments for the script instead?
Thanks.
Application environment is not specified here, can you elaborate more about your application and its environment.
Also did you tried application -R "#$readgroup"
I think eval "application -R '$readgroup'" would do literally the same as you wrote in correct command line sytax, since it gets the whole command as a string.
On the other hand there might be something tricky going on. I think inside application it can't be decided if $readgroup was specified with or without quotes. At last the main() will only get unquoted char *argv[]!
I get the error executing bash syntax error near unexpected token `('
I know the error is caused by the ')' but I thought placing the commands in-between ' ' is suppose to allow the parenthesis in a directory name. How can I fix this without renaming the name?
The matlab / octave code is:
syscmd=strcat({'bash -c '},{''''},{'cd '},dirpathpls,newdirname,{' && exec bash xfade.sh'},{''''}) %used to run script to join files in stretch directory
system(syscmd);
and it produces what is below:
bash -c 'cd /tmp/h1/clients/04212015142432811_Fs_1000_ahh/pls/03sox_a_Fs_1000_ahh_(000_bit)_(0.0000
0sig_in_deg)_to_(508_bit)_(30.00000sig_in_deg) && exec bash xfade.sh'
please note:
It's being called from inside octave 3.8.1 a math program like matlab
Using ' within a bash command line does allow the use of reserved characters like ( without escaping; however, that is not what you are doing. Everything within your 's is being passed to bash for interpretation, bash isn't interpreting the 's as part of the command. Something like this should work:
syscmd=strcat({'bash -c '},{''''},{'cd "'},dirpathpls,newdirname,{'" && exec bash xfade.sh'},{''''}) %used to run script to join files in stretch directory
system(syscmd);
I don't know matlab/octave, but I hope that conveys the idea. The " should effectively escape the parens. The only pitfall there is if your directory name might have a $ or " in it, in that case, or you have ' AND " in your dir name, things are going to get silly.
As I told you in your other question on this topic: Don't use bash -c; it's not necessary for octave to run an external command, and you're doing nothing but making your life harder by trying.
command=strcat({'cd '''},
strrep(strcat(dirpathpls,newdirname),
'''',
'''"''"'''''),
{''' && exec bash xfade.sh'})
system(syscmd);
Two key differences:
We're using the sh -c implicitly created by the system() call
We're escaping the filenames, preventing any malicious content within them from escaping the quotes and being executed.
How that escaping works:
Single-quoted strings in POSIX shells are ended only by a following single-quote. To insert a literal single quote into them, one needs to end the single-quoted string and then enter a different quoting type. Thus:
'"'"'
...in which the first ' ends the prior quoting type; the " enters a double-quoted context (in which a single-quote literal can be recognized; the ' after it is then your literal single-quote character; the " following ends the double-quoted context, and the final ' resumes a single-quoted context.
This is all made more complicated by doubling the 's to ''s for Octave's syntax; this is how one gets
strrep(content, '''', '''"''"''''')
...to replace all 's with '"'"'s.
I have to write a script that generates and executes a command with a variable number of arguments containing file names. These file names may contain spaces, and everything must work fine with or without spaces.
For example, this generated command may look like this :
curl --data-urlencode "js_code#/tmp/some folder/data.txt" http://www.someurl.com
If I use a hard coded command and execute it it all runs fine, with and without spaces. If I create the command text in a string variable however, and execute the string contents, it seems the command does not take the quotes into account, using only the first part of the file :
The script (simplified, just imagine the command string is created using complex rules) :
#!/bin/sh
#prepare
command="curl --data-urlencode \"param_value#/tmp/some folder/data.txt\" www.someurl.com"
#execute
$command
The results :
$ ./test.sh
Warning: Couldn't read data from file ""param_value#/tmp/some", this makes an
Warning: empty POST.
curl: (6) Couldn't resolve host 'folder'
I tried different things, switching quotes style, using things like exec, but I could'nt get this to work.
Any help would be appreciated
Thanks
Note : I should add all this testing is done on Cygwin. It may be important regarding path syntax.
You should use eval :
eval "$command"
Have you tried:
command='curl --data-urlencode "param_value#/tmp/some\ folder/data.txt" www.someurl.com'