so I have a GET command retrieving data from a server, and only need specific parts of this info.
I have a working script but the awk part is very long and I was wondering if I could get some help shortening it.
Current script:
curl --insecure -u $HMCUSER:$HMCPASS -H "Accept: */*" -H "Content-type: application/json" -s --header "X-API-Session:$COOKIE" -X GET https://$HMCIP:6794$BCID1/blades | awk -F\" '{print $50" M1:"$42"\n"$114" M1:"$106"\n"$18" M1:"$10"\n"$98" M1:"$90"\n"$34" M1:"$26"\n"$82" M1:"$74"\n"$66" M1:"$58"\n"$130" M1:"$122}' > ~walkers/blade-info-new
echo -e "\n`cat blade-info-new`\n"
and the output is:
/api/blades/394a7ea8-02d4-11e1-b71a-5cf3fcad1a40 M1:B.1.01
/api/blades/749f35cc-02d7-11e1-946a-5cf3fcad1ef8 M1:B.1.02
/api/blades/eeae9670-02d5-11e1-a5ee-5cf3fcad21e0 M1:B.1.03
/api/blades/3949f5a0-02d4-11e1-85df-5cf3fcad1dc8 M1:B.1.04
/api/blades/d25df328-02d3-11e1-a1e9-5cf3fcad2158 M1:B.1.05
/api/blades/bbecebd8-02d0-11e1-aca7-5cf3fcacf4a0 M1:B.1.06
/api/blades/3016b5d8-02d7-11e1-a66f-5cf3fcad1dd0 M1:B.1.07
/api/blades/75796586-02ea-11e1-8ab0-5cf3fcacf040 M1:B.1.08
(there are two columns: /api/blades/... and M1:B.1.0#)
So I tried this:
for i in {10..130..8}
do
try=$(curl --insecure -u $HMCUSER:$HMCPASS -H "Accept: */*" -H "Content-type: application/json" -s --header "X-API-Session:$COOKIE" -X GET https://$HMCIP:6794$BCID1/blades | awk -v i=$i -F\" '{print $i}')
echo "$try"
done
hoping to get the same output as above and instead I just get the complete JSON object:
{"blades":[{"status":"operating","name":"B.1.03","type":"system-x","object-uri":"/api/blades/eeae9670-02d5-11e1-a5ee-5cf3fcad21e0"},{"status":"operating","name":"B.1.05","type":"system-x","object-uri":"/api/blades/d25df328-02d3-11e1-a1e9-5cf3fcad2158"},{"status":"operating","name":"B.1.01","type":"system-x","object-uri":"/api/blades/394a7ea8-02d4-11e1-b71a-5cf3fcad1a40"},{"status":"operating","name":"B.1.07","type":"system-x","object-uri":"/api/blades/3016b5d8-02d7-11e1-a66f-5cf3fcad1dd0"},{"status":"operating","name":"B.1.06","type":"system-x","object-uri":"/api/blades/bbecebd8-02d0-11e1-aca7-5cf3fcacf4a0"},{"status":"operating","name":"B.1.04","type":"system-x","object-uri":"/api/blades/3949f5a0-02d4-11e1-85df-5cf3fcad1dc8"},{"status":"operating","name":"B.1.02","type":"system-x","object-uri":"/api/blades/749f35cc-02d7-11e1-946a-5cf3fcad1ef8"},{"status":"operating","name":"B.1.08","type":"system-x","object-uri":"/api/blades/75796586-02ea-11e1-8ab0-5cf3fcacf040"}]}
So I was wondering how to get the variable to work? I've been on many websites and everyone seems to say awk -v i=$i should work...
EDIT: The sequence I want to print is the object uri (i.e. /api/blades/...) followed by the blade name (i.e. B.1.01). These infos are all in the JSON object returned by the curl command starting with the tenth field and every 8th field after that (using " as a delimiter):
{"blades":[{"status":"operating","name":"B.1.03","type":"system-x","object-uri":"/api/blades/eeae9670-02d5-11e1-a5ee-5cf3fcad21e0"},{"status":"operating","name":"B.1.05","type":"system-x","object-uri":"/api/blades/d25df328-02d3-11e1-a1e9-5cf3fcad2158"},{"status":"operating","name":"B.1.01","type":"system-x","object-uri":"/api/blades/394a7ea8-02d4-11e1-b71a-5cf3fcad1a40"},{"status":"operating","name":"B.1.07","type":"system-x","object-uri":"/api/blades/3016b5d8-02d7-11e1-a66f-5cf3fcad1dd0"},{"status":"operating","name":"B.1.06","type":"system-x","object-uri":"/api/blades/bbecebd8-02d0-11e1-aca7-5cf3fcacf4a0"},{"status":"operating","name":"B.1.04","type":"system-x","object-uri":"/api/blades/3949f5a0-02d4-11e1-85df-5cf3fcad1dc8"},{"status":"operating","name":"B.1.02","type":"system-x","object-uri":"/api/blades/749f35cc-02d7-11e1-946a-5cf3fcad1ef8"},{"status":"operating","name":"B.1.08","type":"system-x","object-uri":"/api/blades/75796586-02ea-11e1-8ab0-5cf3fcacf040"}]}
The blade names don't have to be in numerical order (B.1.01 to B.1.08), only on the same line as the corresponding ID
EDIT 2: Found a work around. Used a C type for loop instead of the normal bash: for (( i=10; i<=130; i+=8 )) instead of for i in {10..130..8}
The proper answer to this question is to ditch awk (even though I love awk) and use a real JSON parser, e.g. the very handy jq tool.
If I understand correctly you're wanting {10..130..8} to expand to give the required series of $i values.
In my version of bash (it's ooooold: 3.2.25) the string {10..130..8} doesn't expand to anything and so the loop is entered with i="{10..130..8}" and so awk uses ${10..130..8} which appears to simplify to $0 (i.e. the whole curl return string). Hence your problem. You can test if this is the case by putting echo $i inside your loop.
You need a better way of getting the series of values you want. You can use "seq" for this (man seq for more info). $( seq 10 8 130 ) should do it.
Further, you can make it so that curl is only called once with something messier like
# Construct the string of fields
for i in $( seq 10 8 130 ); do
fields="$fields,\$$i"
done
fields=$( echo "$fields" | sed 's/^,//' ) # Remove the leading comma
...curl command... | awk '{print '$fields'}'
I think you want awk to access the updated value of the i variable. Because the awk instruction is between apostrophes (''), the value of i is hidden from awk, but you can avoid it by removing the apostrophes from the piece of the instruction that is replaced by the actual value of i. It is explained on this online AWK manual, in the section Dynamic Variables.
So for your particular case, you could try
awk -F\" '{print $'$i'}'
instead of
awk -v i=$i -F\" '{print $i}'
at the end of your pipeline command.
How about changing record separator (RS), and defining field separator to quote. Then save the name, and print it together with object-url. The command below should give you a starting point
curl [...] | awk -v RS=, -F\" '{
if ($2 ~ /^name$/) {name=$4}
if ($2 ~ /^object-uri$/) {print name, $4}
}'
p.s. Remote the if's and printing $0 if you want to see how the RS=, helps you.
Related
I am trying to use a bash script for Pushgateway and Prometheus, but can't get it working.
I have Pushgateway on my raspberry and Prometheus on another raspberry. All work correctly and I test a simple bash script, this script works correctly.
My simple script (test.sh):
echo "metric 99" | curl --data-binary #- http://localhost:9091/metrics/job/some_job
Now I want to write a more complex script. This script must push to prometheus the metric from CPU usage (the same as the "$ps aux" command or same as "$ top" command). But this script doesn't work and I don't know what to do to modify it..
My more complex script :
#!/bin/bash
z="ps aux"
while read -r $z
do
var=$var$(awk '{print "cpu_usage{process=\""$11"\", pid=\""$2"\"}", $3$z}');
done <<< "$z"
curl -X POST -H "Content-type: text/plain" --data "$var" http://localhost:9091/metrics/job/top/instance/machine
If anyone could help me. Thanks a lot.
I also try this code :
#!/bin/bash
z="ps aux"
while read -r "ps aux"
do
var=$var$(awk '{print "cpu_usage{process=\""$11"\", pid=\""$2"\"}", $3$z}');
done <<< "$z"
curl -X POST -H "Content-type: text/plain" --data "$var" http://localhost:9091/metrics/job/top/instance/machine
But I am not sure about the syntax. What's wrong ?
I try the code :
load=$(ps aux | awk '{ print "cpu_usage{ process=\"" $11 "\",pid=\"" $2 "\"}," $3 }')
curl -X POST -H --data "$load" http://localhost:9091/metrics/job/top/instance/machine
But it doesn't work. The first line is ok, but when i run this code, I find an error message to the curl command:
curl: (3) URL using bad/illegal format or missing URL
========== The solution to my problem is : ==========
ps aux | awk '$3>0 {print "cpu_usage"$2" "$3""}' | curl --data-binary #- http://localhost:9091/metrics/job/top/instance/machine
This command could transfer to the pushgateway all process data with % CPU > 0. In this line, $3 = %CPU, $2 = PID. Be carefull with special caracters. If the result command is an error message, maybe it is because there is special caracter...
If your problem is too complex, divide it into smaller, more manageable pieces and see what they do. Start by analysing output from the awk part.
AWK can be a bit of a handful.
Try a simpler approach:
ps aux | tr -s ' ' ',' | cut -d, -f2,11 |
while read pid process; do
req="cpu_usage{process=$process,pid=$pid}"
echo "Sending CURL Request with this data: $req"
curl -X POST -H "Content-type: text/plain" --data "$req" http://localhost:9091/metrics/job/top/instance/machine
done
You may want to review the brackets. I have no means of testing this.
You seem to be confused about several details of basic Bash syntax.
command <<<"string"
simply passes the literal string as standard input to command. The syntax you seem to be looking for is the process substitution
command < <(other)
which runs other and passes its output as input to command. But this too is an overcomplication; you probably want a simpler straight pipeline.
load=$(ps aux | awk '{ print "cpu_usage{ process=\"" $11 "\",pid=\"" $2 "\"}," $3 }')
curl -X POST -H "Content-type: text/plain" --data "$load" http://localhost:9091/metrics/job/top/instance/machine
where I had to resort to some wild speculation around what you hoped the Awk script should do.
Also, the argument to read is the name of a variable, so read z, not read $z (and usually use read -r unless you specifically require the odd legacy behavior of read with backslashes in its input).
Finally, you basically never want to store a command in a variable; see https://mywiki.wooledge.org/BashFAQ/050
Going forward, probably try http://shellcheck.net/ before asking for human assistance.
/!\ The solution to my problem is : /!\
ps aux | awk '$3>0 {print "cpu_usage"$2" "$3""}' | curl --data-binary #- http://localhost:9091/metrics/job/top/instance/machine
This command could transfer to the pushgateway all process data with % CPU > 0. In this line, $3 = %CPU, $2 = PID. Be carefull with special caracters. If the result command is an error message, maybe it is because there is special caracter...
Thanks...
I am trying to write a program (either python or bash) to extract saved passwords from the chromium web browser.
What I've found so far, the following command extracts the web site address, username and password:
$ sqlite3 ~/.config/chromium/Default/"Login Data" "select * from logins" | cut -d "|" -f 1,4,6
The data however is displayed on the screen and not neatly organised. What I am trying to do is store the data neatly.
The output is at the moment delimited by a | , instead I'd like this delimited by a tab (\t)
I'd like to prefix a header on top
The first column has http:// or https:// preceding the website address, I'd like that stripped.
If possible I'd like to be able to contain all that in a single script. If bash is ill suited for this, I am willing to try different programming language as required.
Any other suggestions are most welcome because I'm doing this as a hobby. The more I get to learn the better.
I am looking for suggestions on what I can do. Upon my last thread I was suggested to post on codereview stack exchange site instead for suggestion. If this is also better suited there forgive my noobness, and if possible tell me how I could move this.
Assuming you saved the results of your query in a file called /path/to/file:
$ cat script
#!/bin/bash
sed -i '1iURL|Username|Password' /path/to/file
sed -i 's|http.*//||g' /path/to/file
awk -F"|" '{ print $1, $2, $3 }' /path/to/file | column -t > /path/to/output
This should do it.
Line1: sed -i '1iURL|Username|Password' /path/to/file: Adds header to your file
Line2: sed -i 's|http.*//||g' /path/to/file: Strips the http*// from your file
Line3: awk -F"|" '{ print $1, $2, $3 }' /path/to/file | column -t > /path/to/output: formats the output to a new file: /path/to/output
How do I extract the variable value of the following line of an html page via Terminal to submit it afterwards via "curl -d" in the same script?
<input type="hidden" name="au_pxytimetag" value="1234567890">
Edit: how do I transfer the extracted value to the "curl -d" command within a single script? might be a silly question, but I'm total noob. =0)
EDITED:
I cannot tell from your question what you are actually trying to do. I originally thought you were trying to extract a variable from a file, but it seems you actually want to firstly, get that file, secondly extract a variable, and thirdly, use variable for something else... so let's address each of those steps:
Firstly you want to grab a page using curl, so you will do
curl www.some.where.com
and the page will be output on your terminal. But actually you want to search for something on that page, so you need to do
curl www.some.where.com | awk something
or
curl www.some.where.com | grep something
But you want to put that into a variable, so you need to do
var=$(curl www.some.where.com | awk something)
or
var=$(curl www.some.where.com | grep something)
The actual command I think you want is
var=$(curl www.some.where.com | awk -F\" '/au_pxytimetag/{print $(NF-1)}')
Then you want to use the variable var for another curl operation, so you will need to do
curl -d "param1=$var" http://some.url.com/somewhere
Original answer
I'd use awk like this:
var=$(awk -F\" '/au_pxytimetag/{print $(NF-1)}' yourfile)
to take second to last field on line containing au_pxytimetag using " as field separator.
Then you can use it like this
curl -d "param1=$var¶m2=SomethingElse" http://some.url.com/somewhere
You can use xmllint:
value=$(xmllint --html --xpath "string(//input[#name='au_pxytimetag']/#value)" index.html)
You can do it with my Xidel:
xidel http://webpage -e "//input[#name='au_pxytimetag']/#value"
But you do not need to.
With
xidel http://webpage -f "(//form)[1]" -e "//what-you-need-from-the-next-page"
you can send all values from the first form on the webpage to the form action and then you can query something from the next page
You can try:
grep au_pxytimetag input.html | sed "s/.* value=\"\(.*\)\".*/\1/"
EDIT:
If you need this on a script:
#!/bin/bash
DATA=$(grep au_pxytimetag input.html | sed "s/.* value=\"\(.*\)\".*/\1/")
curl http://example.com -d $DATA
I am having a little trouble with using a variable and printing the 2nd field with the awk command. I am attempting to grab a number from a value in a file. The value in the file looks like
MAX=10000 (I want the Number only), I am passing this into a variable in a script so in the script I have variables
parm_file=ParmFiles/Parmfile.parm
session=s_session_value
OLD_MAX_SEQ_NR=`awk -F '=' "/$session/ {getline; print $2}" < $parm_file`
because I have double quotes to identify the $session variable, it is taking the $2 as a variable too, and so it is just printing the whole line, instead of the second field.
I've tried also to pass the variable into the awk command like
OLD_MAX_SEQ_NR=`awk -F '=' \
-v var="$session" \
'/var/ {getline; print $2}' < $parm_file`
But it does not seem to be putting the variable where var is. I have even tried hard coding the -v var="s_session_value" and it does not work.
I can't figure out a way to make the command look at the $2 as it normally does instead of a variable. Any help would be greatly appreciated.
Try this:
parm_file=ParmFiles/Parmfile.parm
session=s_session_value
OLD_MAX_SEQ_NR=$(
awk -F'=' -v pat="$session" \
'$0 ~ pat {getline; print $2}' < "$parm_file"
)
You need to pass shell variables to awk by defining an awk variable using -v.
Using variable inside /../ is taken as literal. So use $0~var_name construct.
Using back-ticks is deprecated. Use command substitution $(..)
Quote your variables.
It's a bit tricky without a sample line of the parm file. But I don't understand why you don't use cut, it makes it much easier?
OLD_MAX_SEQ_NR=$(grep "$session" "$parm_file" | cut -d= -f2)
I am trying to do a very basic thing, which I though I should be able to manage all on my own, but alas..
What I am attempting to do, is to get hold of the values returned by "df -h", and to be able to store these values in a bash script array. The purpose is to combine it with other data (like the current timestamp) and output this to a new file/overwrite an existing file.
The following two commands give me what I want on the commandline, but I cannot for the life of me load the output into an array that I can iterate over.
The followng gives me the disk utilization in percent for each disk:
df -h | grep -vE "^Filesystem|shm|boot" | awk '{ print +$5 }'
And the following gives me the disk path:
df -h | grep -vE "^Filesystem|shm|boot" | awk '{ print $6 }'
However, I am unable to assign the output of this into a variable inside a shell script that I can iterate over.
ANY ideas and help would be very much appreciated!!
You can assign the output to an array like this:
arr=(`df -h | grep -vE "^Filesystem|shm|boot" | awk '{ print +$5 }'`)
Then you can iterate over it for example like this:
for var in "${arr[#]}"
do
# things
done
You can use this.
#!/bin/bash
arr=(`df -h | grep -vE "^Filesystem|shm|boot" | awk '{ print +$5 }'`)
for v in "${arr[#]}"
do
echo $v
done
(Note only works with bash, not /bin/sh)