Why is '\' invalid in this command called with os/exec? - bash

When I execute this code written in Go:
package main
import ( "fmt"
"os/exec"
)
func donde(num string) string {
cmd := fmt.Sprintf("wget -qO- \"https://www.pasion.com/contactos-mujeres/%s.htm?edadd=18&edadh=30\"|grep -av \"https:\"|grep -av \"contactos\"|grep -av \"javascript\"|grep -av \"href=\\"/\"", num)
out, err := exec.Command("bash","-c",cmd).Output()
if err != nil {
return fmt.Sprintf("Failed to execute command: %s", cmd)
}
return string(out)
}
func main() {
chicas := map[string][]string{ "Alexia":{"600080000"},
"Paola":{"600070008", "600050007", "600000005", "600000001", "600004", "600000000"}}
for k, v := range chicas {
fmt.Printf("%s\n", k)
for index := range v {
c := donde(v[index])
exec.Command("bash", "-c", c)
fmt.Println(c)}
}
}
I get:
./favoritas.go:8:189: invalid operation: "wget -qO- \"https://www.pasion.com/contactos-mujeres/%s.htm?edadd=18... / "" (operator / not defined on untyped string)
./favoritas.go:8:190: invalid character U+005C '\'
grep -av \"href=\\"/\" seems to be the culprit. Interestingly, similar Python code
works just fine:
from subprocess import run
v = "600000005"
dnd = run('wget -qO- \"https://www.pasion.com/contactos-mujeres/'+v+'.htm?edadd=18&edadh=30\" |grep -av \"https:\"|grep -av \"contactos\"|grep -av \"javascript\" |grep -av \"href=\\"/\"' , capture_output=True, shell=True, text=True, encoding='latin-1').stdout
print(dnd)
and wget -qO- "https://www.pasion.com/contactos-mujeres/600000003.htm?edadd=18&edadh=30" |grep -av "https:"|grep -av "contactos"|grep -av "javascript" |grep -av "href=\"/" executed from my shell (I use Bash) works fine as well.
Why cannot I accomplish the same in my code Go? How might I resolve this issue?
P.S. What is pasted here are just snippets of more lengthy programs.

escaping quotes within a language within a language is hard. Use alternate syntax when available to alleviate this pain.
Your syntax is complex because you chose to enquote the string with double quotes, but the string contains double quotes, so they must be escaped. Additionally, you have double quotes within the string that themselves must be escaped. You've escaped them, but made a typeo in your escaping at the end:
"wget -qO- \"https://www.pasion.com/contactos-mujeres/%s.htm?edadd=18&edadh=30\"|grep -av \"https:\"|grep -av \"contactos\"|grep -av \"javascript\"|grep -av \"href=\\"/\""
you escaped the backslash, but did not include an additional backslash to escape the quote. So the quoted string ended. The / is not enquoted in the string, thus applied to the quoted string as an operator. But string has no / operator, hence the error.
`wget -qO- "https://www.pasion.com/contactos-mujeres/%s.htm?edadd=18&edadh=30"|grep -av "https:"|grep -av "contactos"|grep -av "javascript"|grep -av 'href="/'`
key takeaway: use backticks when appropriate to enquote strings that contain quotes, then you won't need to escape quotes within the string.
additionally, if you use single quote in bash, it will disable all special characters until another single quote is found. grep -av 'href="/' is more straightforward, no?
key takeaway: use single quotes in bash, when appropriate, to delineate literal strings
Better yet, don't shell out unless you really have to
all your pain here is because you took code that was valid in bash, and tried to encapsulate it within another programming language. don't do that unless you really have to.
consider an alternative here that might make your life easier:
Make the http request with Go's net/http library instead of wget.
Parse the HTML in the response with https://pkg.go.dev/golang.org/x/net/html which will be more robust than grep. HTML content does not grep well.

Related

Variable not expanding in Double quotes for bash script

I have a bash script where i'm trying to call a curl which is having a variable value as input. When trying to execute the bash script the variable value is not getting expanded in double quotes.
Expected curl in script after variable expansion should be as following:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id="fgsd34sff334" '
Getting executed as follows when observed in debug mode:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\""$idp_sub"\" '
My bash script is as follows:
#!/bin/bash
idp_sub=""
for idp_sub in $(cat /opt/SP/jboss/home/mayur/es_idp_sub.txt)
do
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\""$idp_sub"\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
done
How does do i expand the variable value within double quotes as shown above in expected output ?
Your double-quoted string is inside single quotes, where it won't be expanded.
Compare:
foo=bar
echo 'foo=\""$foo\"'
echo 'foo="'"$foo"'"'
In the second example, we end the single quotes, and double-quote $foo, then start new single quotes for the final '.
It's probably easier to read if we expand using printf instead:
printf 'foo=%s\n' "$foo"
That's something you might want to run as a process substitution.
BUT...
This is a wrong and dangerous way to construct an SQL query (and the web server is also poor, if it forwards arbitrary queries - I hope it has no write permissions to the data). Read about "SQL command injection" and come back to this code when you understand the issues.
Nothing inside single quotes will be expanded by bash, including any double-quotes, and variable names. The good news is you can end your single-quoted section and immediately start a double-quoted section to introduce the variable, and it will all be concatenated into a single argument for the application (curl). Try:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\"'"$idp_sub"'\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
You can make your code strongly injection-proof by rejecting any string containing a double-quote, but you might reject some strings that have been legitimately escaped.
If you can use the q syntax to quote the string, you can make it more injection-proof, but I guess the attacker just has to inject ]":
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=q\"['"$idp_sub"]'\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
You could then search for and reject the pattern string ]" as your anti-injection, which will allow a much wider class of legitimate strings. You would have to tell the users that you have applied q[] quoting to their input, so they don't have to.

Can't execute ffprobe command

I try to get video file duration with ffprobe. When I run this code I get error:
exit status 1:
var out bytes.Buffer
var stderr bytes.Buffer
cmdArgs := []string{"-i", "bunny.mp4", "-show_entries", "format=duration", "-v", "quiet", "-of", `csv="p=0"`}
cmd := exec.Command("ffprobe", cmdArguments...)
cmd.Stdout = &out
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
fmt.Println(fmt.Sprint(err) + ": " + stderr.String())
}
fmt.Printf("command output: %q\n", out.String())
But when I pass argruments without -of csv="p=0" like this:
cmdArgs := []string{"-i", "bunny.mp4", "-show_entries", "format=duration", "-v", "quiet"}
It works and returns the result (but in bad format):
command output: "[FORMAT]\nduration=3.008000\n[/FORMAT]\n"
So what is the problem and how to solve it ?
Try formatting the argument like this (use double quotes for the string instead of backticks and remove the inner double quotes):
cmdArgs := []string{..., "csv=p=0"}
The Go exec package does not invoke the sytem shell to process the arguments, so you do not need to take the same precautions when specifying them. In this case, there's no need to wrap the portion after the first "=" in quotes.
From the package documentation:
Unlike the "system" library call from C and other languages, the
os/exec package intentionally does not invoke the system shell and
does not expand any glob patterns or handle other expansions,
pipelines, or redirections typically done by shells. The package
behaves more like C's "exec" family of functions. To expand glob
patterns, either call the shell directly, taking care to escape any
dangerous input, or use the path/filepath package's Glob function. To
expand environment variables, use package os's ExpandEnv.

How to exec command to do sed

I have the following snippet that is not working. Compiles but does not do what it is supposed to. Executing the same command on bash works. Why?
hash:="4ab32de"
cmd = "sed -i -e 's/clt_[0-9a-z]*/clt_"+hash+"/g' /tmp/test.env"
parts = strings.Fields(cmd)
for _, part :=range parts {
fmt.Printf("\n%s",part)
}
head = parts[0]
out, err = exec.Command(head,parts[1:]...).Output()
fmt.Printf("\nnew cmd is %s\n",cmd)
fmt.Printf("out:%s",string(out))
The output of parts is perfect, like this
sed
-i
-e
's/clt_[0-9a-z]*/clt_4ab32de/g'
/tmp/test.env
The exec package doesn't use the shell, so take out all quotes (and escapes). In your case, remove the single quotes.

bash script - unable to set variable with double quotes in value

Need help in fixing this bash script to set a variable with a value including double quotes. Somehow I am defining this incorrectly as my values foo and bar are not enclosed in double quotes as needed.
My script thus far:
#!/usr/local/bin/bash
set -e
set -x
host='127.0.0.1'
db='mydev'
_account="foo"
_profile="bar"
_version=$1
_mongo=$(which mongo);
exp="db.profile_versions_20170420.find({account:${_account}, profile:${_profile}, version:${_version}}).pretty();";
${_mongo} ${host}/${db} --eval "$exp"
set +x
Output shows:
+ host=127.0.0.1
+ db=mydev
+ _account=foo
+ _profile=bar
+ _version=201704112004
++ which mongo
+ _mongo=/usr/local/bin/mongo
+ exp='db.profile_versions_20170420.find({account:foo, profile:bar, version:201704112004}).pretty();'
+ /usr/local/bin/mongo 127.0.0.1/mydev --eval 'db.profile_versions_20170420.find({account:foo, profile:bar, version:201704112004}).pretty();'
MongoDB shell version: 3.2.4
connecting to: 127.0.0.1/mydev
2017-04-22T15:32:55.012-0700 E QUERY [thread1] ReferenceError: foo is not defined :
#(shell eval):1:36
What i need is account:"foo", profile:"bar" to be enclosed in double quotes.
In bash (and other POSIX shells), the following 2 states are equivalent:
_account=foo
_account="foo"
What you want to do is to preserve the quotations, therefore you can do the following:
_account='"foo"'
Since part of what you're doing here is forming JSON, consider using jq -- which will guarantee that it's well-formed, no matter what the values are.
host='127.0.0.1'
db='mydev'
_account="foo"
_profile="bar"
_version=$1
json=$(jq -n --arg account "$_account" --arg profile "$_profile" --arg version "$_version" \
'{$account, $profile, version: $version | tonumber}')
exp="db.profile_versions_20170420.find($json).pretty();"
mongo "${host}/${db}" --eval "$exp"
This makes jq responsible for adding literal quotes where appropriate, and will avoid attempted injection attacks (for instance, via a version passed in $1 containing something like 1, "other_argument": "malicious_value"), by replacing any literal " in a string with \"; a literal newline with \n, etc -- or, with the | tonumber conversion, failing outright with any non-numeric value.
Note that some of the syntax above requires jq 1.5 -- if you have 1.4 or prior, you'll want to write {account: $account, profile: $profile} instead of being able to write {$account, $profile} with the key names inferred from the variable names.
When you need to use double quotes inside a double quoted string, escape them with backslashes:
$ foo="acount:\"foo\"" sh -c 'echo $foo'
acount:"foo"
I needed to enquote something already in a variable and stick that in a variable. Expanding on Robert Seaman's answer, I found this worked:
VAR='"'$1'"'
(single quote, double quote, single quote, variable,single quote, double quote, single quote)

How to preserve single and double quotes in shell script arguments WITHOUT the ability to control how they pass

I need to receive arguments I have no control over into a shell script, and preserve any single or double quotes. For instance, a script that simply outputs the given arguments should act as follows:
> my_script.sh "double" 'single' none
"double" 'single' none
I don't have the privilege of augmenting the arguments such as in:
> my_script.sh \"double\" \'single\' none
or
> my_script.sh '"double"' "'single'" none
And neither "$#" nor "$*" work.
I also thought of reading from STDIN and try something like:
> echo "double" 'single' none | my_script.sh
thinking it may help, but no breakthrough so far.
Any suggestions?
CSH / PERL solutions are welcomed.
This is not possible (without escaping), because the shell processes the arguments and removes the quotes before your script is called. As a result, your script does not know about the quotes specified on the command line.
You cannot recover the single/double quotes exactly as they were, because the shell 'eats' them. If you need to call some other script from your script, you can e.g. single quote the arguments again. Here is a PERL solution I use:
sub args2shell
{
local (#argv) = #_;
local $" = '\' \'';
local (#margv);
#margv = map { s/'/'\\''/g; $_ } #argv;
return "\'#margv\'" if #margv;
return undef;
}
Example usage:
$args = args2shell #ARGV;
open F, "find -follow $args ! -type d |";
...

Resources