I have a Jenkins freestyle job (yeah I know..) that sometimes needs to wait for another job with the same parameter to finish if it's running (or not run at all if it failed).
Using curl and the Jenkins API, is it possible to query a certain job and get the status of the last build where certainParam=certainValue?
(The reason I'm asking how to do that with curl is because it doesn't seem to be possible to do in freestyle job and the job can't be migrated to pipeilnes yet.. It seems like curl would be the east way..)
Thanks ahead!
So far I know there's no direct way to achieve it.
I wrote recursive script that search by the values on each build from the very last until match the information.
It prints every build url and the result of the query.
Dependencies
jq - install "jq.x86_64", Command-line JSON processor
Script
#!/bin/bash
user="admin"
pwd="11966e4dd8c33a730abf88e98fb952ebc3"
builds=$(curl -s --user $user:$pwd localhost:8080/job/test/api/json)
urls=$(echo $builds | jq '.builds[] | .url')
while read -r url
do
url=$(echo $url | sed -nr 's/"([^|]*)"/\1/p')
# get the build log
build=$(curl -s --user $user:$pwd "${url}api/json")
# transform the log in a simple structure
build=$(echo $build | jq '.result as $result | .actions[].parameters[]? | select(.name == "certainParam") | {(.name): .value, "result": $result}')
# check if the parameter value and the build result are the expected
result=$(echo $build | jq 'if .certainParam == "certainValue" and .result == "SUCCESS" then true else false end')
# print the result of each build
echo "url=${url}api/json;result=$result"
if [[ $result == true ]]
then
break
fi
done <<< $urls
Result
sh jenkins.sh
url=http://localhost:8080/job/test/12/api/json;result=false
url=http://localhost:8080/job/test/11/api/json;result=true
Related
I am trying to pass the GitHub branch creating API URL as a string into a function to get the status code. But as I tried in many ways, it is not working as I think,
the original URL is :
curl -s -X POST -u [user]:[token] -d '{"ref": "refs/heads/feature/bar", "sha": "'$SHA'"}' https://api.github.com/repos/$user/$repo/git/refs
what I am trying to do is, taking some part of this URL and passing into a function as a string as follows:
new_branch_creating_url="-X POST -u $username:$password -d '{"'ref'": "'refs/heads/'$new_branch_to_be_created''", "'sha'": ""$old_sha_value""}' https://api.github.com/repos/$username/$repository_name/git/refs"
My intention is to get the status code of that URL... and for that my function is
#get status code
get_status_code(){
test_url=$1
status_code=$(curl -s -I $test_url | awk '/HTTP/{print $2}')
#echo "status code :$status_code"
if [ $status_code == 404 ]
then
echo "wrong credentials passed..."
exit 1
else
return $status_code
fi
}
and while debugging the code, I am getting like
++ curl -s -I -X POST -u myusername:tokenid -d ''\''{ref:' refs/heads/branch2, sha: '1b2hudksjahhisabkhsd6asdihds8dsajbsualhcn}'\''' https://api.github.com/repos/myusername/myrepo/git/refs
++ awk '/HTTP/{print $2}'
and also my doubt is why sometimes I am received a wrong status code from the function above which I used to get a status code?
while debugging my code:
status_code=
+ '[' == 404 ']'
git_branches.sh: line 154: [: ==: unary operator expected
+ return
+ new_branch_status_code=2
+ echo 'new branch status code ... 2'
new branch status code ... 2
+ '[' 2 == 200 ']'
actually, it is nothing there on status_code from the function, but I received status code =2,,,
not only this time, but I also received 153 instead of 409... why it is like that?
I know this is not a relevant to ask but I have no choice and also it would be very helpful if someone helps me in the early stage of my learning in the shell scripting...
thank you...
Instead of curl -s -I, use :
curl -I -s -o /dev/null -w "%{http_code}"
which will give directly http_code.
You don't need awk
I am using the below Jenkinsfile. It is printing the value of upload_id when it is executing with the jq command, but when printing the value it is showing null.
Please help me to fix this variable issue.
sh label: '', script: 'upload_id=$(cat output_file.json | jq -r \'.upload_id\')'
sh "echo \"$upload_id\""**
Output :
[Pipeline] sh
cat output_file.json
jq -r .upload_id
upload_id=8f304c6d-804b-440a-xxxx**
[Pipeline] sh
echo upload_id : null
upload_id : null
[Pipeline] }
I would strongly recommend to go with Matt's answer, because it is the cleanest way.
Anyway, there are situations where there is no other choice than to use the shell, so here is the shell way:
script {
def upload_id = sh label: '',
script: 'echo $(cat output_file.json | jq -r \'.upload_id\')',
returnStdout: true
upload_id = upload_id.trim() // remove extraneous whitespace
sh "echo \"$upload_id\""
}
I've already linked to a more detailed answer of mine but you were probably not getting it to work, because you are using a declarative pipeline. In declarative pipeline, you have to use a script block to be able to store return values of steps.
You can easily do this intrinsically in Jenkins Pipeline to avoid all of the issues with subprocess execution. First, read in and parse the JSON file:
upload_info = readJSON(file: 'output_file.json')
Then, you can access the returned values in the assigned upload_info Map normally:
upload_id = upload_info['upload_id']
Jenkins pipeline I need to execute the shell command and the result is the value of def variable.
What shall I do? Thank you
def projectFlag = sh("`kubectl get deployment -n ${namespace}| grep ${project} | wc -l`")
//
if ( "${projectFlag}" == 1 ) {
def projectCI = sh("`kubectl get deployment ${project} -n ${namespace} -o jsonpath={..image}`")
echo "$projectCI"
} else if ( "$projectCI" == "${imageTag}" ) {
sh("kubectl delete deploy ${project} -n ${namespaces}")
def redeployFlag = '1'
echo "$redeployFlag"
if ( "$projectCI" != "${imageTag}" ){
sh("kubectl set image deployment/${project} ${appName}=${imageTag} -n ${namespaces}")
}
else {
def redeployFlag = '2'
}
I believe you're asking how to save the result of a shell command to a variable for later use?
The way to do this is to use some optional parameters available on the shell step interface. See https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script for the documentation
def projectFlag = sh(returnStdout: true,
script: "`kubectl get deployment -n ${namespace}| grep ${project} | wc -l`"
).trim()
Essentially set returnStdout to true. The .trim() is critical for ensuring you don't pickup a \n newline character which will ruin your evaluation logic.
I have a script where I extract values from curl statements using jq. This script works well whenever the if-statement receives one true value, however, it breaks as soon as a second media package is found and has to be processed.
The code:
#!/bin/bash
source mh_auth.sh
rm -f times.txt
touch times.txt
#Read lines from file with mediapackage-id's that are on schedule.
while read LINE; do
curl --silent --digest -u $mh_username:$mh_password -H "X-Requested-Auth: Digest" -H "X-Opencast-Matterhorn-Authorization: true" "$mh_server/workflow/instances.json?mp=$LINE" > $LINE-curl.txt
#Format the file to make it more readable if you need to troubleshoot
/usr/bin/python -m json.tool $LINE-curl.txt > $LINE-curl-final.txt
#Test to see if this mediapackage has been published yet, and if it has, extract the necessary values for calculation
if grep -q -e 'Cleaning up"' $LINE-curl-final.txt; then
echo "Media Package found"
workflows=$( jq '.workflows.workflow[]' < $LINE-curl-final.txt )
for i in "${workflows}"
do
echo "Getting the end_time variable"
end_time=`echo ${i} | jq '.operations.operation[] | select(.description == "Cleaning up") | .completed'`
echo "Done getting end time variable"
echo "Getting ingest time variable"
ingest_time=`echo ${i} | jq '.operations.operation[] | select(.description == "Ingest") | .completed'`
echo "Done getting ingest time variable"
echo $ingest_time $end_time >> times.txt
echo last >> times.txt
done
else
echo "Media Package not published yet"
fi
rm -f $LINE-curl.txt
rm -f $LINE-curl-final.txt
done < scheduled-mediapackages.txt
A successful run yields the following:
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package found
Getting the end_time variable
Done getting end time variable
Getting ingest time variable
Done getting ingest time variable
Whenever I add a second published media package containing exactly the same json as the first to my list, I get the following:
Media Package not published yet
Media Package not published yet
Media Package found
Getting the end_time variable
Done getting end time variable
Getting ingest time variable
Done getting ingest time variable
Media Package found
Getting the end_time variable
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
Done getting end time variable
Getting ingest time variable
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
Done getting ingest time variable
Any ideas how to fix this? I have been going around in circles and cannot get it to work?
I know it might seem arrogant to answer my own question, but for all those out there who has struggled like me, I wanted to add a fix. Before I get any slams for the code maybe being clumsy and "too much", just remember, I just started coding in bash, and this is working. Others might find this and with their experience be able to clean it up. For me though, this is what worked, so I stuck with it.
Thanks to Mark Setchell, I went back to the drawing board to figure out why the while loop was giving me a hard time. I figured out that some of the json return I got had more objects in it than others, so the jq command had to be adjusted to fit each command. I am only posting the relevant code for extracting the data I was looking for in my OP.
if grep -q -e 'Cleaning up"' $LINE-curl-final.txt; then
Because some media packages fail on first processing, then gets "picked up" to be re-processed, they have more than one workflow object. This if-statement searches for the succeeded one between them,and extracts the data with the adjusted jq query
get_count=$( jq -r '.[] | .totalCount' < $LINE-curl-final.txt )
if [[ "$get_count" -gt 1 ]]; then
state=$( jq -r '.workflows.workflow[] | select(.state == "SUCCEEDED" ) | .operations.operation[] | select(.description == "Cleaning up") | .state' < $LINE-curl-final.txt )
if [ "$state" = "SUCCEEDED" ];then
end_time=$( jq '.workflows.workflow[] | select(.state == "SUCCEEDED" ) | .operations.operation[] | select(.description == "Cleaning up") | .completed' < $LINE-curl-final.txt )
ingest_time=$( jq '.workflows.workflow [] | .operations.operation[] | select(.description == "Ingest") | .completed' < $LINE-curl-final.txt )
fi
else
state=$( jq -r '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .state' < $LINE-curl-final.txt )
if [ "$state" = "SUCCEEDED" ];then
end_time=$( jq '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .completed' < $LINE-curl-final.txt )
ingest_time=$( jq '.workflows.workflow.operations.operation[] | select(.description == "Ingest") | .completed' < $LINE-curl-final.txt )
fi
fi
Note the difference between
jq '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .completed'
and
jq '.workflows.workflow[] | select(.state == "SUCCEEDED" ) | .operations.operation[] | select(.description == "Cleaning up") | .completed'
I hope that this idea might help anybody else struggling with the same problem.
I have a Bash script that builds a string to run as a command
Script:
#! /bin/bash
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
illcommando="$serverbin include='$include' server::team_l_start = '${teamAComm}' server::team_r_start = '${teamBComm}' CSVSaver::save='true' CSVSaver::filename = 'out.csv'"
echo "running: $illcommando"
# $illcommando > server-output.log 2> server-error.log
$illcommando
which does not seem to supply the arguments correctly to the $serverbin.
Script output:
running: /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv'
rcssserver-14.0.1
Copyright (C) 1995, 1996, 1997, 1998, 1999 Electrotechnical Laboratory.
2000 - 2009 RoboCup Soccer Simulator Maintenance Group.
Usage: /usr/local/bin/rcssserver [[-[-]]namespace::option=value]
[[-[-]][namespace::]help]
[[-[-]]include=file]
Options:
help
display generic help
include=file
parse the specified configuration file. Configuration files
have the same format as the command line options. The
configuration file specified will be parsed before all
subsequent options.
server::help
display detailed help for the "server" module
player::help
display detailed help for the "player" module
CSVSaver::help
display detailed help for the "CSVSaver" module
CSVSaver Options:
CSVSaver::save=<on|off|true|false|1|0|>
If save is on/true, then the saver will attempt to save the
results to the database. Otherwise it will do nothing.
current value: false
CSVSaver::filename='<STRING>'
The file to save the results to. If this file does not
exist it will be created. If the file does exist, the results
will be appended to the end.
current value: 'out.csv'
if I just paste the command /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv' (in the output after "runnning: ") it works fine.
You can use eval to execute a string:
eval $illcommando
your_command_string="..."
output=$(eval "$your_command_string")
echo "$output"
I usually place commands in parentheses $(commandStr), if that doesn't help I find bash debug mode great, run the script as bash -x script
don't put your commands in variables, just run it
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
PWD=$(pwd)
teamAComm="$PWD/a.sh"
teamBComm="$PWD/b.sh"
include="$PWD/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include=$include server::team_l_start = ${teamAComm} server::team_r_start=${teamBComm} CSVSaver::save='true' CSVSaver::filename = 'out.csv'
./me casts raise_dead()
I was looking for something like this, but I also needed to reuse the same string minus two parameters so I ended up with something like:
my_exe ()
{
mysql -sN -e "select $1 from heat.stack where heat.stack.name=\"$2\";"
}
This is something I use to monitor openstack heat stack creation. In this case I expect two conditions, an action 'CREATE' and a status 'COMPLETE' on a stack named "Somestack"
To get those variables I can do something like:
ACTION=$(my_exe action Somestack)
STATUS=$(my_exe status Somestack)
if [[ "$ACTION" == "CREATE" ]] && [[ "$STATUS" == "COMPLETE" ]]
...
Here is my gradle build script that executes strings stored in heredocs:
current_directory=$( realpath "." )
GENERATED=${current_directory}/"GENERATED"
build_gradle=$( realpath build.gradle )
## touch because .gitignore ignores this folder:
touch $GENERATED
COPY_BUILD_FILE=$( cat <<COPY_BUILD_FILE_HEREDOC
cp
$build_gradle
$GENERATED/build.gradle
COPY_BUILD_FILE_HEREDOC
)
$COPY_BUILD_FILE
GRADLE_COMMAND=$( cat <<GRADLE_COMMAND_HEREDOC
gradle run
--build-file
$GENERATED/build.gradle
--gradle-user-home
$GENERATED
--no-daemon
GRADLE_COMMAND_HEREDOC
)
$GRADLE_COMMAND
The lone ")" are kind of ugly. But I have no clue how to fix that asthetic aspect.
To see all commands that are being executed by the script, add the -x flag to your shabang line, and execute the command normally:
#! /bin/bash -x
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include="$include" server::team_l_start="${teamAComm}" server::team_r_start="${teamBComm}" CSVSaver::save='true' CSVSaver::filename='out.csv'
Then if you sometimes want to ignore the debug output, redirect stderr somewhere.
For me echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}'
was working fine but unable to store output of command into variable.
I had same issue I tried eval but didn't got output.
Here is answer for my problem:
cmd=$(echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}')
echo $cmd
My output is now 20200824