Strange behavior when parsing result from curl + awk - bash

Using curl on ubuntu I am trying to fetch the Jenkins version inspired by:
https://wiki.jenkins.io/display/JENKINS/Remote+access+API
In a bash script I do:
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -Fi X-Jenkins: | awk '{print $2}')
echo "__A__[${VERSION}]__B__"
But when I run the script I get:
]__B__2.89.2
So for some reason the prefix: __A__[ gets swallowed and the suffix gets turned into a prefix.
I have also tried to trim the output with:
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -Fi X-Jenkins: | awk '{print $2}' | sed -e 's/^[ \t]*//')
But it gives the same result.
As suggested below I have also tried with:
echo '__A__['"${VERSION}"']__B__'
But still gives the same/wrong result.
A few other things I have tried (giving the same result)
Same/wrong output
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -i X-Jenkins: | awk '{print $2}')
echo '__A__['"${VERSION}"']__B__'
Same/wrong output
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep X-Jenkins: | awk '{print $2}')
echo '__A__['"${VERSION}"']__B__'
Based on below suggestion I have now tried:
echo $VERSION|od -ax
Which gives:
0000000 2 . 8 9 . 2 cr nl
2e32 3938 322e 0a0d
0000010
If I compare that with:
VERSION_TEMP="2.89.2"
echo $VERSION_TEMP|od -ax
I get:
0000000 2 . 8 9 . 2 nl
2e32 3938 322e 000a
0000007
So looks like its the cr in the VERSION var that is causing the issue (not sure how that explains the whole reversing of prefix/suffix as described above).
SOLVED: Based on input from Romeo I now got it to work with adding |tr -d '\r' :
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep X-Jenkins: | awk '{print $2}'|tr -d '\r')

Apparently the output contains a DOS carriage return.
Try adding tr -d '\015':
version=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" "$URL" |
tr -d '\015' |
awk 'tolower($0) !~ /x-jenkins:/{print $2}')
echo "__A__[$version]__B__"
Uppercase variable names are reserved for system use, so I changed yours to lower case, too, and removed the useless grep.

Related

Jenkins with curl and grep into pipe

Having some big issues with my Jenkins pipeline to make it work. In short tried a lot of combinations, this one seems to be the closest i can get.
def command = $/"curl -v -u ${nexusUser}:${nexusPassword} -X 'GET' \"${nexus_api}\" -H 'accept:application/json' | grep downloadUrl | head -n1 | cut -d : -f 2,3 | tr -d \\\", | xargs curl -O"/$
println(command)
res = sh(returnStdout: true, script: command).trim()
sh("echo ${res}")
All variables are expanded with some escaping done however im getting : not found . Manually trying the below output directly in the container where jenkins is executing this part works fine without issues, so it's not a matter that curl is not found.
curl -u *****:**** -X 'GET' "https://**************/service/rest/v1/search/assets?sort=version&direction=desc&repository=apt-bionic-release&name=*********" -H 'accept:application/json' | grep downloadUrl | head -n1 | cut -d : -f 2,3 | tr -d \", | xargs curl -O: not found
Any thoughts what i could try ?
Thanks!

Timer in a loop - Bash

I have a token whos expire in 30 minutes, i want do renew this token until the 30 minutes time expire, i have the following script calling an api:
function token {
gen_bearer_token=$(curl -X POST "https://api.foo.bar/oauth2/token" -H "accept: application/json" -H "Content-Type: application/x-www-form-urlencoded" -d "client_id=foo&client_secret=bar" | cut -d '{' -f2 | cut -d '}' -f1)
bearer_token=$(echo $gen_bearer_token | awk '{print $2}' | cut -d '"' -f2 | cut -d '"' -f1)
token_type=$(echo $gen_bearer_token | awk '{print $6}' | cut -d '"' -f2 | cut -d '"' -f1)
}
echo -e "HOSTNAME;LAST_SEEN" > ${file}
ids=$(curl -s -X GET 'https://api.foo.bar/devices/queries/devices-scroll/v1?limit=5000' -H 'accept: application/json' -H 'authorization: '${token_type}' '${bearer_token}'' | jq .resources[])
for id in ${ids}
do
result=$(curl -s -X GET 'https://api.foo.bar/devices/entities/devices/v1?ids='${id}'' -H 'accept: application/json' -H 'authorization: '${token_type}' '${bearer_token}'' | jq '.resources[] | "\(.hostname) \(.last_seen)"')
if [[ ! -z ${result} ]]
then
hostname=$(echo ${result} | awk '{print $1}' | cut -d '"' -f2)
last_seen=$(echo ${result} | awk '{print $2}' | cut -d '"' -f1)
echo -e "${hostname};${last_seen}" >> ${file}
fi
done
in this looping, time duration is more than 2 hours (variable duration) + i want to create a timer to renew token if the times pass through 30 minutes, if passed 30 minutes the request in api will fail.
for id in ${ids}
do
result=$(curl -s -X GET 'https://api.foo.bar/devices/entities/devices/v1?ids='${id}'' -H 'accept: application/json' -H 'authorization: '${token_type}' '${bearer_token}'' | jq '.resources[] | "\(.hostname) \(.last_seen)"')
if [[ ! -z ${result} ]]
then
hostname=$(echo ${result} | awk '{print $1}' | cut -d '"' -f2)
last_seen=$(echo ${result} | awk '{print $2}' | cut -d '"' -f1)
echo -e "${hostname};${last_seen}" >> ${file}
fi
done
bash as a suitable timer built-in: the SECONDS variable.
Its value is its initial value plus the number of seconds since the most recent assignment. At shell start-up, it is initialized to 0.
All you need to do is check the value of $SECONDS before using the token, and if its value is greater than 1800 (the number of seconds in 30 minutes), get a new one.
SECONDS=1801 # Make sure a token will be generated before the first call
for id in $ids; do
if [ "$SECONDS" -gt 1800 ]; then
token
SECONDS=0
fi
...
done
I Solved this problem creating a check response with status code.
response=$(curl -H 'authorization: '${token_type}' '${bearer_token}'' --write-out '%{http_code}' --silent --output /dev/null https://api.foo.bar/devices/entities/devices/v1?ids='0000000000000')
if [ ${response} -eq 200 ]; then
do something
else
token
fi

curl sends empty json on content-type: application/json

I am sending cpu and memory usage on json format from Ubuntu to Node.js API using curl with POST method. However the json data is empty at Node.js server.
Bash script on Ubuntu
top -b -n 2 -d 0.5 > top.txt
cpu_i=$(grep Cpu top.txt | cut -d ',' -f 4 | cut -d ' ' -f 2)
cpu=$(echo $cpu_i | cut -d ' ' -f 2)
echo $cpu
mem=$(grep "KiB Mem :" top.txt | cut -d ':' -f 2)
#echo $mem
mem_used=$(echo $mem | cut -d ',' -f 3 | cut -d ' ' -f 2)
echo $mem_used
curl --header "Content-Type: application/json" -d "{\"cpu\":\"$cpu\", \"memory\":\"$mem_used\",\"device\":\"ubuntu\"}" http://192.168.10.10:4000/collector
Output at Node.js server
{}
Remote Address: ::ffff:192.168.10.5
If I remember correctly, the httpverb is set to get by default. If you intend to post it, use -X POST. That will probably solve your problem, because the curl command is OK.
curl -X POST --header "Content-Type: application/json" -d "{\"cpu\":\"$cpu\", \"memory\":\"$mem_used\",\"device\":\"ubuntu\"}" http://192.168.10.10:4000/collector

Curl command in shell script returning error {"Error":"bpapigw-300 Cannot authorize access to resource"

I am trying to execute curl using this shell script:
#!/bin/bash
curl -k -H "Content-Type:application/json" -d '{"username":"admin","password":"adminpw", "tenant":"master"}' https://localhost/tron/api/v1/tokens > /tmp/token.data
grep -Po '{\"token\":\"\K[^ ]...................' /tmp/token.data > /tmp/token
tokendetails=`cat /tmp/token`
for token in $tokendetails
do
TOKEN=`echo $token`
done
userdetails=`cat /tmp/curloutput.txt | sed 's/{"clientInactivityTime"/\n{"clientInactivityTime"/g' | sed 's/\(.*\).\("firstName":[^,]*\)\(.*\)\("lastName":[^,]*\)\(.*\)\("email":[^,]*\)\(.*\)\("username":[^,]*\)\(.*\)/\2,\4,\6,\8/g' | grep username`
for user in $userdetails
do
firstName=`echo $user | sed 's/,/\n/g' | grep firstName | sed 's/.*:"\([^"]*\).*/\1/g'`
lastName=`echo $user | sed 's/,/\n/g' | grep lastName | sed 's/.*:"\([^"]*\).*/\1/g'`
email=`echo $user | sed 's/,/\n/g' | grep email | sed 's/.*:"\([^"]*\).*/\1/g'`
username=`echo $user | sed 's/,/\n/g' | grep username | sed 's/.*:"\([^"]*\).*/\1/g'`
curl -k -X POST "https://haxsne09/tron/api/v1/users" -H "accept: application/json" -H "Authorization: Bearer =${TOKEN}" -H "Content-Type: application/x-www-form-urlencoded" -d "first_name=${firstName}\&last_name=${lastName}\&email=${email}\&password=Tata123^\&username=${username}\&is_active=true"
echo $RESPONSE
done
I am getting ths error:
{"Error":"bpapigw-300 Cannot authorize access to resource: Could not authorize path for user identifier: Failed to get Roles for identifier: REST operation failed 0 times: '[GET /api/v1/current-user][401] currentUserListUnauthorized \u0026{Detail:Invalid token}'. This user is unauthenticated?"}
Do I need to add any syntax before executing curl -k -X POST?
What I see is that -H "Authorization: Bearer =${TOKEN}" contains an = sign which shouldn't be there...
It should be: -H "Authorization: Bearer ${TOKEN}"
More, in a command you use /tmp/curloutput.txt file, which is never created by your script...
The Authorization header you are using is not working. Maybe the syntax is not Bearer =aAbBcCdDeEfF0123456 but something else for the server running on haxsne09, maybe without the = like #MarcoS suggests.
Alternatively, your grep command may be returning one too many characters (a rogue quote maybe).
I rewrote your code below to be more readable. You will notice that I:
Changed your matching groups in sed to capture only the needed parts and put them in variables using read. I also used the -E flag to avoid having to use \( and \)
Removed the useless for loops
Quoted all variable expansions properly
Added some line breaks for readability
Removed some temporary files and associated useless uses of cat
Here is the updated script:
#!/bin/bash
curl -k -H 'Content-Type:application/json' -d \
'{"username":"admin","password":"adminpw", "tenant":"master"}' \
https://localhost/tron/api/v1/tokens > /tmp/token.data
token=$(grep -Po '{"token":"\K[^ ]...................' /tmp/token.data)
IFS=, read -r firstName lastName email username < <(
</tmp/curloutput.txt sed 's/{"clientInactivityTime"/\n&/' |
sed -nE 's/.*."firstName":"([^"]*)".*"lastName":"([^"]*)").*"email":"([^"]*).*"username":"([^"]*)".*/\1,\2,\3,\4/p'
)
curl -k -X POST 'https://haxsne09/tron/api/v1/users' -H 'accept: application/json' \
-H "Authorization: Bearer $token" -H "Content-Type: application/x-www-form-urlencoded" -d \
"first_name=$firstName&last_name=$lastName&email=$email&password=Tata123^&username=$username&is_active=true"
echo

Can not to use POST in CURL

I am trying to make a script to use POST instead of GET but I not success at it.
This is the script:
#!/usr/bin/bash
echo "Content-type: application/json"
echo ""
login=`echo "$QUERY_STRING" | awk '{split($0,array,"&")} END{print array[1]}' | awk '{split($0,array,"=")} END{print array[2]}'`
clave=`echo "$QUERY_STRING" | awk '{split($0,array,"&")} END{print array[2]}' | awk '{split($0,array,"=")} END{print array[2]}'`
if curl -H "Content-Type: application/json" -d "username="$login"&passcode="$clave"" -X POST http://x.x.x.x/abc -s | grep -q mnp;
then
echo 1
else
echo 3
fi
the script works but it still using GET.
screenshot of HTTP request
Any ideas about the right format of the Curl command?
Regards

Resources