Having some big issues with my Jenkins pipeline to make it work. In short tried a lot of combinations, this one seems to be the closest i can get.
def command = $/"curl -v -u ${nexusUser}:${nexusPassword} -X 'GET' \"${nexus_api}\" -H 'accept:application/json' | grep downloadUrl | head -n1 | cut -d : -f 2,3 | tr -d \\\", | xargs curl -O"/$
println(command)
res = sh(returnStdout: true, script: command).trim()
sh("echo ${res}")
All variables are expanded with some escaping done however im getting : not found . Manually trying the below output directly in the container where jenkins is executing this part works fine without issues, so it's not a matter that curl is not found.
curl -u *****:**** -X 'GET' "https://**************/service/rest/v1/search/assets?sort=version&direction=desc&repository=apt-bionic-release&name=*********" -H 'accept:application/json' | grep downloadUrl | head -n1 | cut -d : -f 2,3 | tr -d \", | xargs curl -O: not found
Any thoughts what i could try ?
Thanks!
Related
I have a bash script as below:
curl -s "$url" | grep "https://cdn" | tail -n 1 | awk -F[\",] '{print $2}'
which is working fine, when i run run it, i able to get the cdn url as:
https://cdn.some-domain.com/some-result/
when i put it as variable :
myvariable=$(curl -s "$url" | grep "https://cdn" | tail -n 1 | awk -F[\",] '{print $2}')
and i echo it like this:
echo "CDN URL: '$myvariable'"
i get blank result. CDN URL:
any idea what could be wrong? thanks
If your curl command produces a trailing DOS carriage return, that will botch the output, though not exactly like you describe. Still, maybe try this.
myvariable=$(curl -s "$url" | awk -F[\",] '/https:\/\/cdn/{ sub(/\r/, ""); url=$2} END { print url }')
Notice also how I refactored the grep and the tail (and now also tr -d '\r') into the Awk command. Tangentially, see useless use of grep.
The result could be blank if there's only one item after awk's split.
You might try grep -o to only return the matched string:
myvariable=$(curl -s "$url" | grep -oP 'https://cdn.*?[",].*' | tail -n 1 | awk -F[\",] '{print $2}')
echo "$myvariable"
Objective:
I'm trying to write a script that will fetch two URLs from a GitHub release page and do something different with each one.
So far:
Here's what I've got so far.
λ curl -s https://api.github.com/repos/mozilla-iot/gateway/releases/latest | grep "browser_download_url.*tar.gz" | cut -d : -f 2,3 | tr -d \"
This will return the following:
"https://github.com/mozilla-iot/gateway/releases/download/0.8.1/gateway-8c29257704ddb021344bdaaa790909a0eacf3293bab94e02859828a6fd9b900a.tar.gz"
"https://github.com/mozilla-iot/gateway/releases/download/0.8.1/node_modules-921bd0d58022aac43f442647324b8b58ec5fdb4df57a760e1fc81a71627f526e.tar.gz"
I want to be able to create some directories, pull in the first one, navigate in the directories from the newly pulled zip after extracting it, and then pull in the second.
fetching the first line is easy by piping the output to head -n1. for solving your problem, you need more than just fetching the first URL of the cURL output. give this a try:
#!/bin/bash
# fetch your URLs
answer=`curl -s https://api.github.com/repos/mozilla-iot/gateway/releases/latest | grep "browser_download_url.*tar.gz" | cut -d : -f 2,3 | tr -d \"`
# get URLs and file names
first_file=`echo "$answer" | grep -Eo '.+?\.tar\.gz' | head -n1 | tr -d " "`
second_file=`echo "$answer" | grep -Eo '.+?\.tar\.gz' | head -n2 | tail -1 | tr -d " "`
first_file_name=`echo "$answer" | grep -Eo '[^/]+?\.tar\.gz' | head -n1 `
second_file_name=`echo "$answer" | grep -Eo '[^/]+?\.tar\.gz' | head -n2 | tail -1`
#echo $first_file
#echo $first_file_name
#echo $second_file_name
#echo $second_file
# download first file
wget "$first_file"
# extracting first one that must be in the current directory.
# else, change the directory first and put the path before $first_file!
tar -xzf "$first_file_name"
# do your stuff with the second file
You can simply pipe the URLs to xargs curl;
curl -s https://api.github.com/repos/mozilla-iot/gateway/releases/latest |
grep "browser_download_url.*tar.gz" |
cut -d : -f 2,3 | tr -d \" |
xargs curl -O
Or if you want to do some more manipulation on each URL, perhaps loop over the results:
curl ... | grep ... | cut ... | tr ... |
while IFS= read -r url; do
curl -O "$url"
: maybe do things with "$url" here
done
The latter could easily be extended to someting like
... | while IFS= read -r url; do
d=${url##*/}
mkdir -p "$d"
( cd "$d"
curl -O "$url"
tar zxf *.tar.gz
# end of subshell means effects of "cd" end
)
done
Using curl on ubuntu I am trying to fetch the Jenkins version inspired by:
https://wiki.jenkins.io/display/JENKINS/Remote+access+API
In a bash script I do:
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -Fi X-Jenkins: | awk '{print $2}')
echo "__A__[${VERSION}]__B__"
But when I run the script I get:
]__B__2.89.2
So for some reason the prefix: __A__[ gets swallowed and the suffix gets turned into a prefix.
I have also tried to trim the output with:
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -Fi X-Jenkins: | awk '{print $2}' | sed -e 's/^[ \t]*//')
But it gives the same result.
As suggested below I have also tried with:
echo '__A__['"${VERSION}"']__B__'
But still gives the same/wrong result.
A few other things I have tried (giving the same result)
Same/wrong output
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -i X-Jenkins: | awk '{print $2}')
echo '__A__['"${VERSION}"']__B__'
Same/wrong output
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep X-Jenkins: | awk '{print $2}')
echo '__A__['"${VERSION}"']__B__'
Based on below suggestion I have now tried:
echo $VERSION|od -ax
Which gives:
0000000 2 . 8 9 . 2 cr nl
2e32 3938 322e 0a0d
0000010
If I compare that with:
VERSION_TEMP="2.89.2"
echo $VERSION_TEMP|od -ax
I get:
0000000 2 . 8 9 . 2 nl
2e32 3938 322e 000a
0000007
So looks like its the cr in the VERSION var that is causing the issue (not sure how that explains the whole reversing of prefix/suffix as described above).
SOLVED: Based on input from Romeo I now got it to work with adding |tr -d '\r' :
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep X-Jenkins: | awk '{print $2}'|tr -d '\r')
Apparently the output contains a DOS carriage return.
Try adding tr -d '\015':
version=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" "$URL" |
tr -d '\015' |
awk 'tolower($0) !~ /x-jenkins:/{print $2}')
echo "__A__[$version]__B__"
Uppercase variable names are reserved for system use, so I changed yours to lower case, too, and removed the useless grep.
I'm trying to extract the db element's value attributes from an xml file using xmllint
My XPath Query needs to navigate to the correct dbtype, which I wish to store in an environment variable
Without an environment variable (hard coded), this command works
echo 'cat //rdbmsinfo/dbtype[#value="sqlserver"]/db/#value' | xmllint --shell "config.xml" | grep -v ">" | cut -f 2 -d "=" | tr -d \ | sed 's/"//g'
I am trying with below command:
echo 'cat //rdbmsinfo/dbtype[#value="$ldb_source_typ"]/db/#value' | xmllint --shell "config.xml" | grep -v ">" | cut -f 2 -d "=" | tr -d \ | sed 's/"//g'
where ldb_source_typ is a variable to get the value from shell parameter as "sqlserver". But the above syntax is not giving any output.
Why isn't this working, and how can I fix it?
My config.xml XML looks like this:
<?xml version="1.0"?>
<rdbmsinfo>
<dbtype value="sqlserver">
<db value="sqlsrv1">
<dbhostip>192.168.0.1</dbhostip>
<dbhostportno>2000</dbhostportno>
<dbusername>sample</dbusername>
<dbpassword>sample</dbpassword>
</db>
<db value="sqlsrv2">
<dbhostip>192.168.0.2</dbhostip>
<dbhostportno>2000</dbhostportno>
<dbusername>sample</dbusername>
<dbpassword>sample</dbpassword>
</db>
</dbtype>
<dbtype value="postgresql">
<db value="postsql1">
</db>
</dbtype>
</rdbmsinfo>
As I just found out, you need to set the environment variable between single quotes ( ' ).
So the following should work:
echo 'cat //rdbmsinfo/dbtype[#value="'$ldb_source_typ'"]/db/#value' | xmllint --shell "config.xml" | grep -v ">" | cut -f 2 -d "=" | tr -d \ | sed 's/"//g'
I was able to accomplish this with the following:
echo "cat //rdbmsinfo/dbtype[#value="ldb_source_type"]/db/#value" | sed 's/ldb_source_type/"'$SQLSERVER'"/g' | xmllint --shell config.xml| grep -v ">" | cut -f 2 -d "=" | tr -d \ | sed 's/"//g'
Which produces the following output:
sqlsrv1
-------
sqlsrv2
I didn't care enough to figure out how to make the value placeholder use an $ symbol, I may come back and edit it eventually.
Edit: here's one with a $, but you need to escape it with a :
echo "cat //rdbmsinfo/dbtype[#value="\$ldb_source_type"]/db/#value" | sed 's/\$ldb_source_type/"'$SQLSERVER'"/g'| xmllint --shell config.xml| grep -v ">" | cut -f 2 -d "=" | tr -d \ | sed 's/"//g'
I have a simple bash script that uses an api to add itself into a database. But the script keeps adding ' ' to my variables and its breaking curl.
hostname=`hostname`
ip_address=`ip add show eth0 | grep 'inet ' | awk '{ print $2 }' | cut -d '/' -f1`
env=`hostname | cut -d '-' -f1`
os=`cat /etc/issue.net | head -1`
date=`date`
curl -H 'Content-Type: application/json' -PUT "https://10.10.10.10/database" -k -d '{"Environment":"'$env'","Hostname":"'$hostname'","IP_Address":"'$ip_address'","OS":"'$os'","State":"OK","Updated_Time":"'$date'"}'
exit $?
The output is this:
curl -H 'Content-Type: application/json' -PUT https://10.10.10.10/database -k -d '{"Environment":"ops","Hostname":"ex-example-host","IP_Address":"10.10.10.10","OS":"Ubuntu' 14.04 'LTS","State":"OK","Updated_Time":"Thu' Aug 14 15:27:55 PDT '2014"}'
Both the $date and $hostname put ' ' on the inside the format breaking the curl. Is there a way to fix this?
Thanks,
The problem is that you are leaving the parameter expansions unquoted in bash, so spaces in those values break up the word being passed to curl. It would be simpler to swap your use of double and single quotes, if JSON allowed you to use single quotes. That not being the case, I'd store the JSON in a variable using read and a here document first to simplify the quoting.
read -r data <<EOF
{"Environment":"$env","Hostname":"$hostname","IP_Address":"$ip_address","OS":"$os","State":"OK","Updated_Time":"$date"}
EOF
curl ... -d "$data"