I have the following snippet that is not working. Compiles but does not do what it is supposed to. Executing the same command on bash works. Why?
hash:="4ab32de"
cmd = "sed -i -e 's/clt_[0-9a-z]*/clt_"+hash+"/g' /tmp/test.env"
parts = strings.Fields(cmd)
for _, part :=range parts {
fmt.Printf("\n%s",part)
}
head = parts[0]
out, err = exec.Command(head,parts[1:]...).Output()
fmt.Printf("\nnew cmd is %s\n",cmd)
fmt.Printf("out:%s",string(out))
The output of parts is perfect, like this
sed
-i
-e
's/clt_[0-9a-z]*/clt_4ab32de/g'
/tmp/test.env
The exec package doesn't use the shell, so take out all quotes (and escapes). In your case, remove the single quotes.
Related
When I execute this code written in Go:
package main
import ( "fmt"
"os/exec"
)
func donde(num string) string {
cmd := fmt.Sprintf("wget -qO- \"https://www.pasion.com/contactos-mujeres/%s.htm?edadd=18&edadh=30\"|grep -av \"https:\"|grep -av \"contactos\"|grep -av \"javascript\"|grep -av \"href=\\"/\"", num)
out, err := exec.Command("bash","-c",cmd).Output()
if err != nil {
return fmt.Sprintf("Failed to execute command: %s", cmd)
}
return string(out)
}
func main() {
chicas := map[string][]string{ "Alexia":{"600080000"},
"Paola":{"600070008", "600050007", "600000005", "600000001", "600004", "600000000"}}
for k, v := range chicas {
fmt.Printf("%s\n", k)
for index := range v {
c := donde(v[index])
exec.Command("bash", "-c", c)
fmt.Println(c)}
}
}
I get:
./favoritas.go:8:189: invalid operation: "wget -qO- \"https://www.pasion.com/contactos-mujeres/%s.htm?edadd=18... / "" (operator / not defined on untyped string)
./favoritas.go:8:190: invalid character U+005C '\'
grep -av \"href=\\"/\" seems to be the culprit. Interestingly, similar Python code
works just fine:
from subprocess import run
v = "600000005"
dnd = run('wget -qO- \"https://www.pasion.com/contactos-mujeres/'+v+'.htm?edadd=18&edadh=30\" |grep -av \"https:\"|grep -av \"contactos\"|grep -av \"javascript\" |grep -av \"href=\\"/\"' , capture_output=True, shell=True, text=True, encoding='latin-1').stdout
print(dnd)
and wget -qO- "https://www.pasion.com/contactos-mujeres/600000003.htm?edadd=18&edadh=30" |grep -av "https:"|grep -av "contactos"|grep -av "javascript" |grep -av "href=\"/" executed from my shell (I use Bash) works fine as well.
Why cannot I accomplish the same in my code Go? How might I resolve this issue?
P.S. What is pasted here are just snippets of more lengthy programs.
escaping quotes within a language within a language is hard. Use alternate syntax when available to alleviate this pain.
Your syntax is complex because you chose to enquote the string with double quotes, but the string contains double quotes, so they must be escaped. Additionally, you have double quotes within the string that themselves must be escaped. You've escaped them, but made a typeo in your escaping at the end:
"wget -qO- \"https://www.pasion.com/contactos-mujeres/%s.htm?edadd=18&edadh=30\"|grep -av \"https:\"|grep -av \"contactos\"|grep -av \"javascript\"|grep -av \"href=\\"/\""
you escaped the backslash, but did not include an additional backslash to escape the quote. So the quoted string ended. The / is not enquoted in the string, thus applied to the quoted string as an operator. But string has no / operator, hence the error.
`wget -qO- "https://www.pasion.com/contactos-mujeres/%s.htm?edadd=18&edadh=30"|grep -av "https:"|grep -av "contactos"|grep -av "javascript"|grep -av 'href="/'`
key takeaway: use backticks when appropriate to enquote strings that contain quotes, then you won't need to escape quotes within the string.
additionally, if you use single quote in bash, it will disable all special characters until another single quote is found. grep -av 'href="/' is more straightforward, no?
key takeaway: use single quotes in bash, when appropriate, to delineate literal strings
Better yet, don't shell out unless you really have to
all your pain here is because you took code that was valid in bash, and tried to encapsulate it within another programming language. don't do that unless you really have to.
consider an alternative here that might make your life easier:
Make the http request with Go's net/http library instead of wget.
Parse the HTML in the response with https://pkg.go.dev/golang.org/x/net/html which will be more robust than grep. HTML content does not grep well.
I am trying to access an env variable in Jenkins pipeline and want to use it in a Shell Script executing in the same pipeline but a differnt step,
pipeline {
agent any
tools {
maven 'M2'
}
environment {
stable_revision = sh(script: 'curl -H "Authorization: Basic $base64encoded" "https://xyz.sad" | jq -r "name"', returnStdout: true)
}
stages {
stage('Initial-Checks') {
steps {
echo "Stable Revision: ${env.stable_revision}" //displays the value
bat "sh && sh undeploy.sh"
}}
...
}}
This is a sample Shell script, it has many lines, but I have an issue in only accessing the above stable_revision variable,
#!/bin/bash
echo xyz = ${stable_revision} #### this gives the right value xyz = 22
echo xyz2 = ${stable_revision}/d ### here after the value the /d is written in new line
For example, let's say the stable_revision value is 22, then in the SH script echo I am getting the value as,
xyz2 = 22
/d
I want the value to be xyz2 = 22/d
You can use .trim() to strip off a trailing newline.
environment {
stable_revision = sh(script: 'curl -H "Authorization: Basic $base64encoded" "https://xyz.sad" | jq -r "name"', returnStdout: true).trim()
}
returnStdout (optional):
If checked, standard output from the task is returned as the step value as a String, rather than being printed
to the build log. (Standard error, if any, will still be printed to
the log.) You will often want to call .trim() on the result to strip
off a trailing newline.
https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script
If you use bash instead of sh for your commands, you can benefit from Bash's built-in string transformations
Here it trims all trailing characters from the [:space:] class witch includes actual spaces and newlines.
echo "xyz2 = ${stable_revision%%[[:space:]]}/d"
If $stable_revision is always an integer, you can force the shell to use it like an integer with:
echo "xyz2 = $((stable_revision))/d"
If you are sure that $stable_revision contains no space, you can force the shell to trim all spaces by using it like a table element:
sr=($stable_revision); echo "xyz2 = ${sr[0]}/d"
You can also use the automatic trimming of a sub-shell returned value, that would trim any leading, trailing and duplicate spaces in-between:
echo "xyz2 = $(echo ${stable_revision})/d"`
Below is a snippet from my Jenkins file -
stage('Configure replication agents') {
environment {
AUTHOR_NAME="XX.XX.XX.XX"
PUBLISHER_NAME="XX.XX.XX.XX"
REPL_USER="USER"
REPL_PASSWORD="PASSWORD"
AUTHOR_PORT="4502"
PUBLISHER_PORT="4503"
AUTHOR="http://${AUTHOR_NAME}:${AUTHOR_PORT}"
PUBLISHER="http://${PUBLISHER_NAME}:${PUBLISHER_PORT}"
S_URI= "${PUBLISHER}/bin/receive?sling:authRequestLogin=1"
}
steps {
sh 'curl -u XX:XX --data "status=browser&cmd=createPage&label=${PUBLISHER_NAME}&title=${PUBLISHER_NAME}&parentPath =/etc/replication/agents.author&template=/libs/cq/replication/templates/agent" ${AUTHOR}/bin/wcmcommand'
}
The above command, in Jenkins console, is printed as
curl -u XX:XX --data status=browser&cmd=createPage&label=XXXX&title=XXX&parentPath =/etc/replication/agents.author&template=/libs/cq/replication/templates/agent http://5XXXX:4502/bin/wcmcommand
Note how the double quotes "" are missing.
I need to preserve the double quotes after --data in this command. How do I do it?
I tried using forward slashes but that didnt work.
Cheers
To expand on my comment, a quick test revealed its the case.
You need to escape twice, once the quote for the shell with a slash, and once that slash with a slash for groovy itself.
node() {
sh 'echo "asdf"'
sh 'echo \"asdf\"'
sh 'echo \\"asdf\\"'
}
Result
[Pipeline] {
[Pipeline] sh
+ echo asdf
asdf
[Pipeline] sh
+ echo asdf
asdf
[Pipeline] sh
+ echo "asdf"
"asdf"
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
After long time of struggling and googling, this is what has worked for me on similar use case:
sh("ssh root#my.server.com \"su user -c \\\"mkdir ${newDirName}\\\"\"")
Update: How I think it gets interpreted
1] sh extension strips first escaping (\" becomes " and \\ becomes \, first and last " are not part of input)
ssh root#my.server.com "su user -c \"mkdir ${newDirName}\""
2] ssh command strips second level of escaping (\" becomes ", while outer " also not part of input)
su user -c "mkdir ${newDirName}"
I had double quotes inside the variable, so escaped single quotes worked for me:
sh "git commit -m \'${ThatMayContainDoubleQuotes}\'"
I needed the output to be with trailing \\ so I had to do something like this
echo 'key1 = \\\\"__value1__\\\\"' > auto.file
File looks like
cat auto.file
key1 = \\"__value1__\\"
Dependent Script
export value1="some-value"
var=${value1}
# Read in template one line at the time, and replace variables
tmpfile=$(mktemp)
sed -E 's/__(([^_]|_[^_])*)__/${\\1}/g' auto.file > ${tmpfile}
while read auto
do
eval echo "$auto"
done < "${tmpfile}" > autoRendered.file
rm -f ${tmpfile}
Rendered File looks like
cat autoRendered.file
key1 = "some-value"
For anyone who comes looking for a fix to a similar issue with quoting numbers during helm install/upgrade, you can use --set-string instead of --set
Ref: https://helm.sh/docs/chart_best_practices/values/#consider-how-users-will-use-your-values
Question
I am willing to pass arguments to a jshell script. For instance, I would have liked something like this:
jshell myscript.jsh "some text"
and then to have the string "some text" available in some variable inside the script.
However, jshell only expects a list of files, therefore the answer is:
File 'some text' for 'jshell' is not found.
Is there any way to properly pass arguments to a jshell script?
Workaround so far
My only solution so far is to use an environment variable when calling the script:
ARG="some test" jshell myscript.jsh
And then I can access it in the script with:
System.getenv().get("ARG")
And what about option -R
> jshell -v -R-Da=b ./file.jsh
for script
{
String value = System.getProperty("a");
System.out.println("a="+value);
}
/exit
will give you
> jshell -v -R-Da=b ./file.jsh
a=b
Another way, would be following:
{
class A {
public void main(String args[])
{
for(String arg : args) {
System.out.println(arg);
}
}
}
new A().main(System.getProperty("args").split(" "));
}
and execution
> jshell -R-Dargs="aaa bbb ccc" ./file_2.jsh
Update
Previous solution will fail with more complex args. E.g. 'This is my arg'.
But we can benefit from ant and it's CommandLine class
import org.apache.tools.ant.types.Commandline;
{
class A {
public void main(String args[])
{
for(String arg : args) {
System.out.println(arg);
}
}
}
new A().main(Commandline.translateCommandline(System.getProperty("args")));
}
and then, we can call it like this:
jshell --class-path ./ant.jar -R-Dargs="aaa 'Some args with spaces' bbb ccc" ./file_2.jsh
aaa
Some args with spaces
bbb
ccc
Of course, ant.jar must be in the path that is passed via --class-path
Oracle really screwed this up, there is no good way to do this. In addition to #mko's answer and if you use Linux(probably will work on Mac too) you can use process substitution.
jshell <(echo 'String arg="some text"') myscript.jsh
And then you can just use arg in myscript.jsh for example:
System.out.println(arg) // will print "some text"
You can simplify it with some bash function and probably write a batch file that will write to a temp file and do the same on windows.
It's completely beyond me how Oracle could ignore this. 8-() But anyway: if your system uses bash as shell, you can combine this approach replacing the shebang with the idea to (ab-)use system properties to transport the whole command line into a variable:
//usr/bin/env jshell --execution local "-J-Da=$*" "$0"; exit $?
String commandline = System.getProperty("a");
System.out.println(commandline);
/exit
This way, you can call the script on the commandline simply adding the arguments: thisscript.jsh arg1 arg2 would print arg1 arg2.
Please note that this joins all parameters into one String, separated by one space. You can split it again with commandline.split("\s"), but please be aware that this isn't exact: there is no difference between two parameters a b and one parameter "a b".
If you have a fixed number of arguments, you can also pass all of these into separate system properties with "-J-Darg1=$1" "-J-Darg2=$1" "-J-Darg3=$1" etc. Please observe that you have to use -R-D... if you are not using --execution local
Another variant is generating the script on the fly with bash's process substitution. You can use such a script also simply as thisscript.jsh arg1 arg2 also on Unix-like systems having a bash.
#!/usr/bin/env bash
jshell <(
cat <<EOF
System.out.println("$1");
System.out.println("$2");
/exit
EOF
)
This allows to access individual parameters, though it will break when there are double quotes or other special characters in a parameter. Expanding on that idea: here's a way to put all parameters into an Java String array, quoting some of those characters:
#!/usr/bin/env bash
set -- "${#//\\/\\\\}"
set -- "${#//\"/\\\"}"
set -- "${#/#/\"}"
set -- "${#/%/\",}"
jshell <(
cat <<EOF
String[] args = new String[]{$#};
System.out.println(Arrays.asList(args));
/exit
EOF
)
The set -- statements double backslashes, quote double quotes and prefix a " and append a ", to transform the arguments into a valid Java array.
Recently, I was inspired by answers from Oleg and Hans-Peter Störr enough to try to combine them so that a) I could use normal shell arguments b) write regular Java code expecting a String[] args input:
//usr/bin/env jshell <(ARGS=; for var in "$#"; do ARGS+="\"$var\","; done; echo "String[] args = {$ARGS}") "$0"; exit $?
System.out.println("RESULT: " + Arrays.asList(args));
/exit
Using Hans' header line and then inlining as demonstrated by Oleg which builds the $# args into a String[] args var.
With that you can chmod +x your script and execute it with regular old arguments:
]$ ./script.jsh foo bar
RESULT: [test, bar]
I'm using Capistrano for deployment and I need to run a command for setting the correct environment in the .htaccess file on the server.
I do that with sed like this: run "sed -i -r 's/APPLICATION_ENV \w+/APPLICATION_ENV #{stage}/' #{current_release}/public/.htaccess"
My problem is that the \w+ is escaped by Ruby/Cap and the regexp ends up trying to match only w+. If I use \\w+ I end up with a regexp that attempts to match \\w+.
How can I have an interpolated double quoted string and successfully escape the \w? Do I really need to change to concatenation of single quoted string and variables?
On my sysem your sample works but you could try it with one of these two (comment the ones out you don't use)
begin
stage = "stage"
current_release = "current_release"
s = "sed -i -r 's/APPLICATION_ENV \\w+/APPLICATION_ENV #{stage}/' #{current_release}/public/.htaccess"
s = "sed -i -r 's/APPLICATION_ENV #{"\\w+"}/APPLICATION_ENV #{stage}/' #{current_release}/public/.htaccess"
s = "sed -i -r 's/APPLICATION_ENV #{92.chr}w+/APPLICATION_ENV #{stage}/' #{current_release}/public/.htaccess"
puts s
system s
rescue
puts $!
system('pause')
end