I have a simple bash script that uses an api to add itself into a database. But the script keeps adding ' ' to my variables and its breaking curl.
hostname=`hostname`
ip_address=`ip add show eth0 | grep 'inet ' | awk '{ print $2 }' | cut -d '/' -f1`
env=`hostname | cut -d '-' -f1`
os=`cat /etc/issue.net | head -1`
date=`date`
curl -H 'Content-Type: application/json' -PUT "https://10.10.10.10/database" -k -d '{"Environment":"'$env'","Hostname":"'$hostname'","IP_Address":"'$ip_address'","OS":"'$os'","State":"OK","Updated_Time":"'$date'"}'
exit $?
The output is this:
curl -H 'Content-Type: application/json' -PUT https://10.10.10.10/database -k -d '{"Environment":"ops","Hostname":"ex-example-host","IP_Address":"10.10.10.10","OS":"Ubuntu' 14.04 'LTS","State":"OK","Updated_Time":"Thu' Aug 14 15:27:55 PDT '2014"}'
Both the $date and $hostname put ' ' on the inside the format breaking the curl. Is there a way to fix this?
Thanks,
The problem is that you are leaving the parameter expansions unquoted in bash, so spaces in those values break up the word being passed to curl. It would be simpler to swap your use of double and single quotes, if JSON allowed you to use single quotes. That not being the case, I'd store the JSON in a variable using read and a here document first to simplify the quoting.
read -r data <<EOF
{"Environment":"$env","Hostname":"$hostname","IP_Address":"$ip_address","OS":"$os","State":"OK","Updated_Time":"$date"}
EOF
curl ... -d "$data"
Related
Example here:
gitrepo=$(jq -r '.gitrepo' 0.json)
releasetag=$(curl --silent ""https://api.github.com/repos/\"$gitrepo\""/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
echo "$releasetag"
Used \" to escape characters.
0.json:
{
"type": "github-releases",
"gitrepo": "ipfs/go-ipfs"
}
How to put $gitrepo to work inside $releasetag?
Thanks in advance!
Bash variables expand inside quoted " strings.
gitrepo="$(jq -r '.gitrepo' 0.json)"
releasetag="$(
curl --silent "https://api.github.com/repos/$gitrepo/releases/latest" \
| grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/'
)"
echo "$releasetag"
Btw, as you are using jq to extract .gitrepo from 0.json, you could also use it in the exact same way to extract .tag_name from curl's output (instead of using grep and sed) like so:
gitrepo="$(jq -r '.gitrepo' 0.json)"
releasetag="$(
curl --silent "https://api.github.com/repos/$gitrepo/releases/latest" \
| jq -r '.tag_name'
)"
echo "$releasetag"
And to simplify it even further (depending on your use case), just write:
curl --silent "https://api.github.com/repos/$(jq -r '.gitrepo' 0.json)/releases/latest" \
| jq -r '.tag_name'
I am sending cpu and memory usage on json format from Ubuntu to Node.js API using curl with POST method. However the json data is empty at Node.js server.
Bash script on Ubuntu
top -b -n 2 -d 0.5 > top.txt
cpu_i=$(grep Cpu top.txt | cut -d ',' -f 4 | cut -d ' ' -f 2)
cpu=$(echo $cpu_i | cut -d ' ' -f 2)
echo $cpu
mem=$(grep "KiB Mem :" top.txt | cut -d ':' -f 2)
#echo $mem
mem_used=$(echo $mem | cut -d ',' -f 3 | cut -d ' ' -f 2)
echo $mem_used
curl --header "Content-Type: application/json" -d "{\"cpu\":\"$cpu\", \"memory\":\"$mem_used\",\"device\":\"ubuntu\"}" http://192.168.10.10:4000/collector
Output at Node.js server
{}
Remote Address: ::ffff:192.168.10.5
If I remember correctly, the httpverb is set to get by default. If you intend to post it, use -X POST. That will probably solve your problem, because the curl command is OK.
curl -X POST --header "Content-Type: application/json" -d "{\"cpu\":\"$cpu\", \"memory\":\"$mem_used\",\"device\":\"ubuntu\"}" http://192.168.10.10:4000/collector
Using curl on ubuntu I am trying to fetch the Jenkins version inspired by:
https://wiki.jenkins.io/display/JENKINS/Remote+access+API
In a bash script I do:
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -Fi X-Jenkins: | awk '{print $2}')
echo "__A__[${VERSION}]__B__"
But when I run the script I get:
]__B__2.89.2
So for some reason the prefix: __A__[ gets swallowed and the suffix gets turned into a prefix.
I have also tried to trim the output with:
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -Fi X-Jenkins: | awk '{print $2}' | sed -e 's/^[ \t]*//')
But it gives the same result.
As suggested below I have also tried with:
echo '__A__['"${VERSION}"']__B__'
But still gives the same/wrong result.
A few other things I have tried (giving the same result)
Same/wrong output
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep -i X-Jenkins: | awk '{print $2}')
echo '__A__['"${VERSION}"']__B__'
Same/wrong output
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep X-Jenkins: | awk '{print $2}')
echo '__A__['"${VERSION}"']__B__'
Based on below suggestion I have now tried:
echo $VERSION|od -ax
Which gives:
0000000 2 . 8 9 . 2 cr nl
2e32 3938 322e 0a0d
0000010
If I compare that with:
VERSION_TEMP="2.89.2"
echo $VERSION_TEMP|od -ax
I get:
0000000 2 . 8 9 . 2 nl
2e32 3938 322e 000a
0000007
So looks like its the cr in the VERSION var that is causing the issue (not sure how that explains the whole reversing of prefix/suffix as described above).
SOLVED: Based on input from Romeo I now got it to work with adding |tr -d '\r' :
VERSION=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" $URL | grep X-Jenkins: | awk '{print $2}'|tr -d '\r')
Apparently the output contains a DOS carriage return.
Try adding tr -d '\015':
version=$(curl -k -i -X GET --insecure --silent --header \"Authorization: Bearer $TOKEN \" "$URL" |
tr -d '\015' |
awk 'tolower($0) !~ /x-jenkins:/{print $2}')
echo "__A__[$version]__B__"
Uppercase variable names are reserved for system use, so I changed yours to lower case, too, and removed the useless grep.
This question already has answers here:
Bash script processing limited number of commands in parallel
(4 answers)
Closed 6 years ago.
I wrote bash script to calculate the hashes of a files, when it's file path given and queries one among the file hash to get the result of that hash from a API service ( Parsing using jq ) .
#Scanner.sh
#!/bin/bash
for i in $(find $1 -type f);do
md5_checksum=$(md5sum $i|cut -d' ' -f 1)
sha1_checksum=$(sha1sum $i|cut -d' ' -f 1)
sha256_checksum=$(sha256sum $i|cut -d' ' -f 1)
json_result="$(curl --silent -H 'Authorization:Basic XXXX' 'https://XXXXX.com/api/databrowser/malware_presence/query/sha1/'$sha1_checksum'?format=json&extended=true'| jq -r '.rl.malware_presence.name,.rl.malware_presence.level' | awk -vORS=, '{print $1}' |sed 's/,$/\n/')"
echo "$md5_checksum,$sha1_checksum,$sha256_checksum,$json_result"
done
#Result :
c63a1576e4b416e6944a1c39dbdfc3b4,fd55dfdd9a432e34ce6c593cd09a48f457f7aab6,e2d1c1975c2406f60f1c0fe5255560c2cd06434e669933536a86296a2eb70020,Malware,5
Now, it's taking too much time to process one and get results for one file hash ( 10 Sec ). How can i send 5 request per second and get the results faster ?
Any suggestions please ?
You could put your code in a function and run it in the background, with something like this:
runCurl() {
md5_checksum=$(md5sum $1|cut -d' ' -f 1)
sha1_checksum=$(sha1sum $1|cut -d' ' -f 1)
sha256_checksum=$(sha256sum $1|cut -d' ' -f 1)
json_result="$(curl --silent -H 'Authorization:Basic XXXX' 'https://XXXXX.com/api/databrowser/malware_presence/query/sha1/'$sha1_checksum'?format=json&extended=true'| jq -r '.rl.malware_presence.name,.rl.malware_presence.level' | awk -vORS=, '{print $1}' |sed 's/,$/\n/')"
echo "$md5_checksum,$sha1_checksum,$sha256_checksum,$json_result"
}
for i in $(find $1 -type f);do
runCurl $i &
done
I need to set or access multiple cURL variables so I can access them later in a script. For example:
curl -s --write-out "%{http_code} | %{local_ip} | %{time_total}" "http://endpoint.com/payload"
Now how can I access http_code or local_ip to do things like add them to an bash array, etc? Is the only option to grep them out of the response?
You can pipe your curl command to a read command :
curl -s --write-out "write-out: %{http_code} | %{local_ip} | %{time_total}\n" "http://yahoo.com" | \
sed -n '/^write-out:/ s///p' | \
while IFS='|' read http_code local_ip time_total;
do
printf "http_code: %s\nlocal_ip: %s\ntotal_time: %s\n" $http_code $local_ip $time_total;
# or in an array
curlvars=($http_code $local_ip $time_total)
for data in "${curlvars[#]}"
do
printf "%s | " $data
done
done
I added a \n to the write-out string to allow process it as a line.
The sed command extract the write-out line from the curl output.
In the read command you can define a separator and assign all parsed strings to vars.