So in my Jenkins pipeline I run a couple of curl commands across different stages. I store the ouput of Stage1 into a file and for every item in that list I run another curl command and use the output of that to extract some values using jq.
However from the second stage I can't seem to store the jq extracted values into variables to echo them later. What am I doing wrong?
{Stage1}
.
.
.
{Stage2}
def lines = stageOneList.readLines()
lines.each { line -> println line
stageTwoList = sh (script: "curl -u $apptoken" + " -X GET --url " + '"' + "$appurl" + "components/tree?component=" + line + '"', returnStdout: true)
pfName = sh (script: "jq -r '.component.name' <<< '${stageTwoList}' ")
pfKey = sh (script: "jq -r '.component.key' <<< '${stageTwoList}' ")
echo "Component Names and Keys\n | $pfName | $pfKey |"
}
returns in the end for Stage2
[Pipeline] sh
+ jq -r .component.name
digital-hot-wallet-gateway
[Pipeline] sh
+ jq -r .component.key
dhwg
[Pipeline] echo
Component Names and Keys
| null | null |
Any help in the right direction appreciated!
You passed true as the argument for the returnStdout argument to the shell step method for stageTwoList, but then forgot to use the same argument for the JSON parsed returns to the next two variable assignments:
def lines = stageOneList.readLines()
lines.each { line -> println line
stageTwoList = sh(script: "curl -u $apptoken" + " -X GET --url " + '"' + "$appurl" + "components/tree?component=" + line + '"', returnStdout: true)
pfName = sh(script: "jq -r '.component.name' <<< '${stageTwoList}' ", returnStdout: true)
pfKey = sh(script: "jq -r '.component.key' <<< '${stageTwoList}' ", returnStdout: true)
echo "Component Names and Keys\n | $pfName | $pfKey |"
}
Note you can also make this much easier on yourself by doing the JSON parsing natively in Groovy and with Jenkins Pipeline step methods:
String stageTwoList = sh(script: "curl -u $apptoken" + " -X GET --url " + '"' + "$appurl" + "components/tree?component=" + line + '"', returnStdout: true)
Map stageTwoListData = readJSON(text: stageTwoList)
pfName = stageTwoListData['component']['name']
pfKey = stageTwoListData['component']['key']
I am trying to execute a shell script on a windows node using Jenkins.
The bash script uses sort -u flag in one of the steps to filter out unique elements from an existing array
list_unique=($(echo "${list[#]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))
Note - shebang used in the script is #!/bin/bash
On calling the script from command prompt as - bash test.sh $arg1
I got the following error -
-uThe system cannot find the file specified.
I understand the issue was that with the above call, sort.exe was being used from command prompt and not the Unix sort command. To get around this I changed the path variable in Windows System variables and moved \cygwin\bin ahead of \Windows\System32
This fixed the issue and the above call gave me the expected results.
However, When the same script is called on this node using Jenkins, I get the same error again
-uThe system cannot find the file specified.
Jenkins stage calling the script
stage("Run Test") {
options {
timeout(time: 5, unit: 'MINUTES')
}
steps {
script {
if(fileExists("${Test_dir}")){
dir("${Test_dir}"){
if(fileExists("test.sh")){
def command = 'bash test.sh ${env.arg1}'
env.output = sh(returnStdout: true , script : "${command}").trim()
if (env.output == "Invalid"){
def err_msg = "Error Found."
sh "echo -n '" + err_msg + " ' > ${ERR_MSG_FILE}"
error(err_msg)
}
sh "echo Running tests for ${env.output}"
}
}
}
}
}
}
Kindly Help
I need Jenkins to run a shell script named create_environment.sh including the following command:
sed -i '' "s~variable \"backendPoolID\" { default = \".*\"~variable \"backendPoolID\" { default = \"$backend_address_pool_id\"~g" var.tf
When I run the script inside the Jenkins machine it works without a problem, but inside the Jenkins pipeline I get the Error:
sed: can't read s~variable "backendPoolID" { default = ".*"~variable "backendPoolID" { default = ""~g: No such file or directory
The pipeline step:
steps {
withCredentials([azureServicePrincipal('xxxx')]) {
dir("${WORKSPACE}/azure/terraform/deployment/${params.AZURE_ENV}") {
echo " *************************** Deploy /${params.AZURE_ENV} ***************************** "
sh "chmod 777 *"
sh "./create_environment.sh"
echo " *************************** Deploy erfolgreich ***************************** "
}
I already tried to replace sh " with sh """, but it didn´t helped.
Do anyone have some experience with that?
I am currently using :
Runtime.getRuntime().exec("open -a Terminal" + directory + " " + argument);
My argument happens to also be a directory.
My problem is that it will open argument instead of considering it an argument of directory. The outcome will be the same as running the following:
Runtime.getRuntime().exec("open -a Terminal" + directory)
and
Runtime.getRuntime().exec("open -a Terminal" + argument)
Instead of typing directly in a terminal :
$/Users/xxxxx/RestOfPath /Users/xxxxx/argument
Is there a solution which would look like this without parenthesis?
Runtime.getRuntime().exec("open -a Terminal (" + directory + " " + argument+ ")");
Thank you!
You could try escaping the space
Runtime.getRuntime().exec("open -a Terminal" + directory + "\\ " + argument);
I am using plink to ssh to many servers and execute commands. But, unable to execute commands on console when password has special characters.
Usage: plink.exe -ssh -pw "password" user#myhost "command to execute"
apof~!##$%^&*()_+{}|:"sdfsdfs!df - Success when escaped " with \.
apof~!##<>$%^&*()_+{}|:"sdfsdfs!df - Success when escaped " with \.
apof~!##$%^&*()_+{}|:"<>sdfsdfs!df - Failed. Escaped " with \. Failure message: > was unexpected at this time.
apof~!##$%^&*()_+{}|:"><sdfsdfs!df - Failed. Escaped " with \. Failure message: < was unexpected at this time.
apof~!##$%^&*()_+{}|:"<sdfsdfs!df - Failed. Escaped " with \. Failure message: The system cannot find the path specified.
apof~!##$%^&*()_+{}|:">sdfsdfs!df - Failed. Escaped " with \. Failure message: The system cannot find the path specified.
Looks like when < or > appear after ", the escape does not work. Many I know how to handle this?
Realized that the issue is not with plink to escape the special characters, but with how cmd handles special characters. For this, I created a simple Java class to take the command line argument and print it:
public class MyTest {
public static void main(String args[]) {
System.out.println(args[0]);
}
}
I passed inputs with special characters to this application and here is the output:
C:\>java MyTest "apof~!##$%^&*()_+{}|:"<>?-=`[]\;',./sss"
> was unexpected at this time.
C:\>java MyTest "apof~!##$%^&*()_+{}|:\"<>?-=`[]\;',./sss"
> was unexpected at this time.
C:\>java MyTest "apof~!##$%^&*()_+{}|:\"^<^>?-=`[]\;',./sss"
apof~!##$%^&*()_+{}|:"<>?-=`[]\;',./sss
C:\>java MyTest "apof~!##$%^&*()_+{}|:\"^<^>?-=`[]\;',./s\"sasfafds"
apof~!##$%^&*()_+{}|:"<>?-=`[]\;',./s"sasfafds
C:\>java MyTest "apof~!##$%^&*()_+{}|:\"^<^>?-=`[]\;',./s\"sasf<>afds"
apof~!##$%^&*()_+{}|:"<>?-=`[]\;',./s"sasf<>afds
C:\>java MyTest "apof~!##$%^&*()_+{}|:\"^<^>?-=`[]\;',./s\"sasf<>af\"ds"
apof~!##$%^&*()_+{}|:"<>?-=`[]\;',./s"sasf<>af"ds
C:\>java MyTest "apof~!##$%^&*()_+{}|:\"^<^>?-=`[]\;',./s\"sasf<>af\"dasd<>fs"
> was unexpected at this time.
C:\>java MyTest "apof~!##$%^&*()_+{}|:\"^<^>?-=`[]\;',./s\"sasf<>af\"dasd^<^>fs"
apof~!##$%^&*()_+{}|:"<>?-=`[]\;',./s"sasf<>af"dasd<>fs
C:\>java MyTest "apof~!#<>#$%^&*()_+{}|:\"^<^>?-=`[]\;',./s\"sasf<>af\"dasd^<^>fs"
apof~!#<>#$%^&*()_+{}|:"<>?-=`[]\;',./s"sasf<>af"dasd<>fs
From this analysis, it is found that only special character to be escaped is " and any < or > symbol occurring after alternate " has also to be escaped with ^. This document pointed me to the escape part.
Below is the python function which I have written for generating the escaped parameter:
def escapepassword(initpassword):
passsplit = list(initpassword)
quotesflag=False
index = -1
for token in passsplit:
index += 1
if quotesflag and (token in ("<", ">")):
passsplit[index] = "^" + token
else:
if token == '"':
passsplit[index] = "\\\""
quotesflag = not quotesflag
return "".join(passsplit)
Still I had issues with calling the executable with escaped arguments. I was using the following way of executing the command:
subprocess.Popen('plink.exe -ssh -batch -pw' + escapepassword(password) + ' ' + username + '#' + ipaddress + ' "' + command + '"', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Or to say in Java, it would be:
Runtime.getRuntime().exec("plink.exe -ssh -batch -pw" + escapepassword(password) + " " + username + "#" + ipaddress + " \"" + command + "\"");
This has given much trouble. The way python or Java execute the process is different. The special characters gets escaped and gets passed to the process. Hence, it creates much more issues. To handle this, I changed the above code to:
commandtoexecute = ['plink.exe', '-ssh', '-batch', '-pw', password, username + '#' + ipaddress, command]
subprocess.Popen(commandtoexecute, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
In Java, could be written as:
Runtime.getRuntime().exec(new String [] {"plink.exe", "-ssh", "-batch", "-pw", password, username + "#" + ipaddress, command);
Here, command had space in between, for e.g. ls -al, but this is the way to pass arguments to start a process. When I tried passing command between ", it had given error saying command not found because " was also getting passed for execution on the remote system.
Note:
I had used the command WMIC path win32_process get name,Commandline | find "plink.exe" to see the commandline arguments. The output is html escaped, however, it has helped to find the arguments passed.