I am trying to do a very basic thing, which I though I should be able to manage all on my own, but alas..
What I am attempting to do, is to get hold of the values returned by "df -h", and to be able to store these values in a bash script array. The purpose is to combine it with other data (like the current timestamp) and output this to a new file/overwrite an existing file.
The following two commands give me what I want on the commandline, but I cannot for the life of me load the output into an array that I can iterate over.
The followng gives me the disk utilization in percent for each disk:
df -h | grep -vE "^Filesystem|shm|boot" | awk '{ print +$5 }'
And the following gives me the disk path:
df -h | grep -vE "^Filesystem|shm|boot" | awk '{ print $6 }'
However, I am unable to assign the output of this into a variable inside a shell script that I can iterate over.
ANY ideas and help would be very much appreciated!!
You can assign the output to an array like this:
arr=(`df -h | grep -vE "^Filesystem|shm|boot" | awk '{ print +$5 }'`)
Then you can iterate over it for example like this:
for var in "${arr[#]}"
do
# things
done
You can use this.
#!/bin/bash
arr=(`df -h | grep -vE "^Filesystem|shm|boot" | awk '{ print +$5 }'`)
for v in "${arr[#]}"
do
echo $v
done
(Note only works with bash, not /bin/sh)
Related
I'm writing a script that checks a list of all the users connected to the server (using who) and writes to the file Information the list of usernames of only those having letters a, b, c or d. This is what I have so far:
who | grep '[a-d]' >> Information
However, the command who displays this:
username pts/148 2019-01-29 16:09 (IP address)
What I don't understand is why my grep search is also displaying the pts/148, date, time, and IP address. I just want it to send the username to the file Information.
Any help is appreciated.
Another way is to use the command cut to get the first part of the string only.
who | cut -f 1 -d ' ' | grep '[a-d]' >> Information
Using awk to output records where the first clumn matches [a-d]:
$ who | awk '$1~/[a-d]/' >> Information
Using grep to search for lines with [a-d] before the first space:
$ who | grep -o "^[^ ]*[a-d][^ ]*" >> Information
You need to get the first word, otherwise grep will display the entire line that has the matching text. You could use awk:
who | awk '{ if (substr($1,1,1) ~ /^[a-d]/ ) print $1 }' >>Information
I am archiving and using split to produce several parts while also printing the output files (from split on STDERR, which I am redirecting to STDOUT). However the loop over the output data doesn't happen until after the command returns.
Is there anyway to actively split over the STDOUT output of a command before it returns?
The following is what I currently have, but it only prints the list of filenames after the command returns:
export IFS=$'\n'
for line in `data_producing_command | split -d -b $CHUNK_SIZE --verbose - $ARCHIVE_PREFIX 2>&1`; do
FILENAME=`echo $line | awk '{ print $3 }'`
echo " - $FILENAME"
done
Try this:
data_producing_command | split -d -b $CHUNK_SIZE --verbose - $ARCHIVE_PREFIX 2>&1 | while read -r line
do
FILENAME=`echo $line | awk '{ print $3 }'`
echo " - $FILENAME"
done
Note however that any variables set in the while loop will not preserve their values after the loop (the while loop runs in a subshell).
There's no reason for the for loop or the read or the echo. Just pipe the stream to awk:
... | split -d -b $CHUNK_SIZE --verbose - test 2>&1 |
awk '{printf " - %s\n", $3 }'
You are going to see some delay from buffering, but unless your system is very slow or you are very perceptive, you're not likely to notice it.
The command substitution needs1 to run before the for loop can start.
for item in $(command which produces items); do ...
whereas a while read -r can start consuming output as soon as the first line is produced (or, more realistically, as soon as the output buffer is full):
command which produces items |
while read -r item; do ...
1 Well, it doesn't absolutely need to, from a design point of view, I suppose, but that's how it currently works.
As William Pursell already noted, there is no particular reason to run Awk inside a while read loop, because that's something Awk does quite well on its own, actually.
command which produces items |
awk '{ print " - " $3 }'
Of course, with a reasonably recent GNU Coreutils split, you could simply do
split --filter='printf " - %s\n" "$FILE"'; cat >"$FILE" ... options
so I have a GET command retrieving data from a server, and only need specific parts of this info.
I have a working script but the awk part is very long and I was wondering if I could get some help shortening it.
Current script:
curl --insecure -u $HMCUSER:$HMCPASS -H "Accept: */*" -H "Content-type: application/json" -s --header "X-API-Session:$COOKIE" -X GET https://$HMCIP:6794$BCID1/blades | awk -F\" '{print $50" M1:"$42"\n"$114" M1:"$106"\n"$18" M1:"$10"\n"$98" M1:"$90"\n"$34" M1:"$26"\n"$82" M1:"$74"\n"$66" M1:"$58"\n"$130" M1:"$122}' > ~walkers/blade-info-new
echo -e "\n`cat blade-info-new`\n"
and the output is:
/api/blades/394a7ea8-02d4-11e1-b71a-5cf3fcad1a40 M1:B.1.01
/api/blades/749f35cc-02d7-11e1-946a-5cf3fcad1ef8 M1:B.1.02
/api/blades/eeae9670-02d5-11e1-a5ee-5cf3fcad21e0 M1:B.1.03
/api/blades/3949f5a0-02d4-11e1-85df-5cf3fcad1dc8 M1:B.1.04
/api/blades/d25df328-02d3-11e1-a1e9-5cf3fcad2158 M1:B.1.05
/api/blades/bbecebd8-02d0-11e1-aca7-5cf3fcacf4a0 M1:B.1.06
/api/blades/3016b5d8-02d7-11e1-a66f-5cf3fcad1dd0 M1:B.1.07
/api/blades/75796586-02ea-11e1-8ab0-5cf3fcacf040 M1:B.1.08
(there are two columns: /api/blades/... and M1:B.1.0#)
So I tried this:
for i in {10..130..8}
do
try=$(curl --insecure -u $HMCUSER:$HMCPASS -H "Accept: */*" -H "Content-type: application/json" -s --header "X-API-Session:$COOKIE" -X GET https://$HMCIP:6794$BCID1/blades | awk -v i=$i -F\" '{print $i}')
echo "$try"
done
hoping to get the same output as above and instead I just get the complete JSON object:
{"blades":[{"status":"operating","name":"B.1.03","type":"system-x","object-uri":"/api/blades/eeae9670-02d5-11e1-a5ee-5cf3fcad21e0"},{"status":"operating","name":"B.1.05","type":"system-x","object-uri":"/api/blades/d25df328-02d3-11e1-a1e9-5cf3fcad2158"},{"status":"operating","name":"B.1.01","type":"system-x","object-uri":"/api/blades/394a7ea8-02d4-11e1-b71a-5cf3fcad1a40"},{"status":"operating","name":"B.1.07","type":"system-x","object-uri":"/api/blades/3016b5d8-02d7-11e1-a66f-5cf3fcad1dd0"},{"status":"operating","name":"B.1.06","type":"system-x","object-uri":"/api/blades/bbecebd8-02d0-11e1-aca7-5cf3fcacf4a0"},{"status":"operating","name":"B.1.04","type":"system-x","object-uri":"/api/blades/3949f5a0-02d4-11e1-85df-5cf3fcad1dc8"},{"status":"operating","name":"B.1.02","type":"system-x","object-uri":"/api/blades/749f35cc-02d7-11e1-946a-5cf3fcad1ef8"},{"status":"operating","name":"B.1.08","type":"system-x","object-uri":"/api/blades/75796586-02ea-11e1-8ab0-5cf3fcacf040"}]}
So I was wondering how to get the variable to work? I've been on many websites and everyone seems to say awk -v i=$i should work...
EDIT: The sequence I want to print is the object uri (i.e. /api/blades/...) followed by the blade name (i.e. B.1.01). These infos are all in the JSON object returned by the curl command starting with the tenth field and every 8th field after that (using " as a delimiter):
{"blades":[{"status":"operating","name":"B.1.03","type":"system-x","object-uri":"/api/blades/eeae9670-02d5-11e1-a5ee-5cf3fcad21e0"},{"status":"operating","name":"B.1.05","type":"system-x","object-uri":"/api/blades/d25df328-02d3-11e1-a1e9-5cf3fcad2158"},{"status":"operating","name":"B.1.01","type":"system-x","object-uri":"/api/blades/394a7ea8-02d4-11e1-b71a-5cf3fcad1a40"},{"status":"operating","name":"B.1.07","type":"system-x","object-uri":"/api/blades/3016b5d8-02d7-11e1-a66f-5cf3fcad1dd0"},{"status":"operating","name":"B.1.06","type":"system-x","object-uri":"/api/blades/bbecebd8-02d0-11e1-aca7-5cf3fcacf4a0"},{"status":"operating","name":"B.1.04","type":"system-x","object-uri":"/api/blades/3949f5a0-02d4-11e1-85df-5cf3fcad1dc8"},{"status":"operating","name":"B.1.02","type":"system-x","object-uri":"/api/blades/749f35cc-02d7-11e1-946a-5cf3fcad1ef8"},{"status":"operating","name":"B.1.08","type":"system-x","object-uri":"/api/blades/75796586-02ea-11e1-8ab0-5cf3fcacf040"}]}
The blade names don't have to be in numerical order (B.1.01 to B.1.08), only on the same line as the corresponding ID
EDIT 2: Found a work around. Used a C type for loop instead of the normal bash: for (( i=10; i<=130; i+=8 )) instead of for i in {10..130..8}
The proper answer to this question is to ditch awk (even though I love awk) and use a real JSON parser, e.g. the very handy jq tool.
If I understand correctly you're wanting {10..130..8} to expand to give the required series of $i values.
In my version of bash (it's ooooold: 3.2.25) the string {10..130..8} doesn't expand to anything and so the loop is entered with i="{10..130..8}" and so awk uses ${10..130..8} which appears to simplify to $0 (i.e. the whole curl return string). Hence your problem. You can test if this is the case by putting echo $i inside your loop.
You need a better way of getting the series of values you want. You can use "seq" for this (man seq for more info). $( seq 10 8 130 ) should do it.
Further, you can make it so that curl is only called once with something messier like
# Construct the string of fields
for i in $( seq 10 8 130 ); do
fields="$fields,\$$i"
done
fields=$( echo "$fields" | sed 's/^,//' ) # Remove the leading comma
...curl command... | awk '{print '$fields'}'
I think you want awk to access the updated value of the i variable. Because the awk instruction is between apostrophes (''), the value of i is hidden from awk, but you can avoid it by removing the apostrophes from the piece of the instruction that is replaced by the actual value of i. It is explained on this online AWK manual, in the section Dynamic Variables.
So for your particular case, you could try
awk -F\" '{print $'$i'}'
instead of
awk -v i=$i -F\" '{print $i}'
at the end of your pipeline command.
How about changing record separator (RS), and defining field separator to quote. Then save the name, and print it together with object-url. The command below should give you a starting point
curl [...] | awk -v RS=, -F\" '{
if ($2 ~ /^name$/) {name=$4}
if ($2 ~ /^object-uri$/) {print name, $4}
}'
p.s. Remote the if's and printing $0 if you want to see how the RS=, helps you.
This question already has answers here:
How to extract one column of a csv file
(18 answers)
Closed 7 years ago.
I am trying to parse in a CSV file which contains a typical access control matrix table into a shell script. My sample CSV file would be
"user","admin","security"
"user1","x",""
"user2","","x"
"user3","x","x"
I would be using this list in order to create files in their respective folders. The problem is how do I get it to store the values of column 2/3 (admin/security)? The output I'm trying to achieve is to group/sort all users that have admin/security rights and create files in their respective folders. (My idea is to probably store all admin/security users into different files and run from there.)
The environment does not allow me to use any Perl or Python programs. However any awk or sed commands are greatly appreciated.
My desired output would be
$ cat sample.csv
"user","admin","security"
"user1","x",""
"user2","","x"
"user3","x","x"
$ cat security.csv
user2
user3
$ cat admin.csv
user1
user3
if you can use cut(1) (which you probably can if you're on any type of unix) you can use
cut -d , -f (n) (file)
where n is the column you want.
You can use a range of columns (2-3) or a list of columns (1,3).
This will leave the quotes but you can use a sed command or something light-weight for that.
$ cat sample.csv
"user","admin","security"
"user1","x",""
"user2","","x"
"user3","x","x"
$ cut -d , -f 2 sample.csv
"admin"
"x"
""
"x"
$ cut -d , -f 3 sample.csv
"security"
""
"x"
"x"
$ cut -d , -f 2-3 sample.csv
"admin","security"
"x",""
"","x"
"x","x"
$ cut -d , -f 1,3 sample.csv
"user","security"
"user1",""
"user2","x"
"user3","x"
note that this won't work for general csv files (doesn't deal with escaped commas) but it should work for files similar to the format in the example for simple usernames and x's.
if you want to just grab the list of usernames, then awk is pretty much the tool made for the job, and an answer below does a good job that I don't need to repeat.
But a grep solution might be quicker and more lightweight
The grep solution:
grep '^\([^,]\+,\)\{N\}"x"'
where N is the Nth column, with the users being column 0.
$ grep '^\([^,]\+,\)\{1\}"x"' sample.csv
"user1","x",""
"user3","x","x"
$ grep '^\([^,]\+,\)\{2\}"x"' sample.csv
"user2","","x"
"user3","x","x"
from there on you can use cut to get the first column:
$ grep '^\([^,]\+,\)\{1\}"x"' sample.csv | cut -d , -f 1
"user1"
"user3"
and sed 's/"//g' to get rid of quotes:
$ grep '^\([^,]\+,\)\{1\}"x"' sample.csv | cut -d , -f 1 | sed 's/"//g'
user1
user3
$ grep '^\([^,]\+,\)\{2\}"x"' sample.csv | cut -d , -f 1 | sed 's/"//g'
user2
user3
Something to get you started (please note this will not work for csv files with embedded commas and you will have to use a csv parser):
awk -F, '
NR>1 {
gsub(/["]/,"",$0);
if($2!="" && $3!="")
print $1 " has both privileges";
print $1 > "file"
}' csv
So I am building a script to check for files with certain errors in a bunch of files, based on output from an SQL DB. The file with the error shall be sent to me via mail.
The problem is that when I try to send the mail, I get the message
"script.sh: 9: mutt: not found" Which does not occur, if I send the mail before the PATH variable is created.
The script looks as following:
JOB=$(sudo cat /tmp/sqltest.txt | awk '{ print $5 }')
DATE=$(sudo cat /tmp/sqltest.txt | awk '{ print $1 }')
CODE=$(sudo cat /tmp/sqltest.txt | awk '{ print $3 }')
PATH=$(grep ${CODE} /tmp/unzip/* | awk '{ print $1 }' | cut -d':' -f1 | head -n 1)
echo "File containing error message for job "${JOB}" at "${DATE}"" | mutt -a "/tmp/sqltest.txt" -s "Mail title" -- <mail#address>
In short, grep finds the file where the error code is, awk picks out the column with the path to the file, the column also comes with a timestamp which cut removes and head ensures that I only get one result, if the error is reported several places.
I can send the mail with mutt if I use it after variable CODE, instead of PATH, though I unfortunately need PATH instead of /tmp/sqltest.txt
Do you have any ideas on what might cause this?
What we got here is a classic case of trying to use an environment variable (and a pretty important one !) : just use another variable name to get rid of the error. As some suggested, it is good practice to try to avoid full-uppercase variables.
Environment Variables
There is a couple of environment variables inside Bash, PATH being one of it.
You can get the list of both environment and shell variables using the set command.
Source :
Environment Variable on Wikipedia
You can simply add the output obtained by command substitution to the PATH.
Change the line setting the PATH to
PATH=$(grep ${CODE} /tmp/unzip/* | awk '{ print $1 }' | cut -d':' -f1
| head -n 1):${PATH}
The change in PATH would be valid for the duration of the script.