Bash error: Integer expression expected - bash

In the sections below, you'll see the shell script I am trying to run on a UNIX machine, along with a transcript.
When I run this program, it gives the expected output but it also gives an error shown in the transcript. What could be the problem and how can I fix it?
First, the script:
#!/usr/bin/bash
while read A B C D E F
do
E=`echo $E | cut -f 1 -d "%"`
if test $# -eq 2
then
I=`echo $2`
else
I=90
fi
if test $E -ge $I
then
echo $F
fi
done
And the transcript of running it:
$ df -k | ./filter.sh -c 50
./filter.sh: line 12: test: capacity: integer expression expected
/etc/svc/volatile
/var/run
/home/ug
/home/pg
/home/staff/t
/packages/turnin
$ _

Before the line that says:
if test $E -ge $I
temporarily place the line:
echo "[$E]"
and you'll find something very much non-numeric, and that's because the output of df -k looks like this:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 954316620 212723892 693109608 24% /
udev 10240 0 10240 0% /dev
: :
The offending line there is the first, which will have its fifth field Use% turned into Use, which is definitely not an integer.
A quick fix may be to change your usage to something like:
df -k | sed -n '2,$p' | ./filter -c 50
or:
df -k | tail -n+2 | ./filter -c 50
Either of those extra filters (sed or tail) will print only from line 2 onwards.
If you're open to not needing a special script at all, you could probably just get away with something like:
df -k | awk -vlimit=40 '$5+0>=limit&&NR>1{print $5" "$6}'
The way it works is to only operate on lines where both:
the fifth field, converted to a number, is at least equal to the limit passed in with -v; and
the record number (line) is two or greater.
Then it simply outputs the relevant information for those matching lines.
This particular example outputs the file system and usage (as a percentage like 42%) but, if you just want the file system as per your script, just change the print to output $6 on its own: {print $6}.
Alternatively, if you do the percentage but without the %, you can use the same method I used in the conditional: {print $5+0" "$6}.

Related

Get a percentage of randomly chosen lines from a text file

I have a text file (bigfile.txt) with thousands of rows. I want to make a smaller text file with 1 % of the rows which are randomly chosen. I tried the following
output=$(wc -l bigfile.txt)
ds1=$(0.01*output)
sort -r bigfile.txt|shuf|head -n ds1
It give the following error:
head: invalid number of lines: ‘ds1’
I don't know what is wrong.
Even after you fix your issues with your bash script, it cannot do floating point arithmetic. You need external tools like Awk which I would use as
randomCount=$(awk 'END{print int((NR==0)?0:(NR/100))}' bigfile.txt)
(( randomCount )) && sort -r file | shuf | head -n "$randomCount"
E.g. Writing a file with with 221 lines using the below loop and trying to get random lines,
tmpfile=$(mktemp /tmp/abc-script.XXXXXX)
for i in {1..221}; do echo $i; done >> "$tmpfile"
randomCount=$(awk 'END{print int((NR==0)?0:(NR/100))}' "$tmpfile")
If I print the count, it would return me a integer number 2 and using that on the next command,
sort -r "$tmpfile" | shuf | head -n "$randomCount"
86
126
Roll a die (with rand()) for each line of the file and get a number between 0 and 1. Print the line if the die shows less than 0.01:
awk 'rand()<0.01' bigFile
Quick test - generate 100,000,000 lines and count how many get through:
seq 1 100000000 | awk 'rand()<0.01' | wc -l
999308
Pretty close to 1%.
If you want the order random as well as the selection, you can pass this through shuf afterwards:
seq 1 100000000 | awk 'rand()<0.01' | shuf
On the subject of efficiency which came up in the comments, this solution takes 24s on my iMac with 100,000,000 lines:
time { seq 1 100000000 | awk 'rand()<0.01' > /dev/null; }
real 0m23.738s
user 0m31.787s
sys 0m0.490s
The only other solution that works here, heavily based on OP's original code, takes 13 minutes 19s.

How to use parameter in conf file for disk usage?

I have a disk script like this:
#!/bin/bash
filesys=(
/
)
[ -f "$(pwd)/filesys.conf" ] && filesys=($(<$(pwd)/filesys.conf))
date=$(date +"%d\/%m\/%Y")
df -P "${filesys[#]}" |
sed -ne 's/^.* \([0-9]\+\)% \(.*\)$/'$date', \2, \1%/p' > disk.log
I have filesys.conf for work which filesystem:
/
/run
And this is output (disk.log):
23/05/2016, /, 78%
23/05/2016, /run, 0%
Question:
I need filesys.conf because server filesystems always change, conf file easy for me. But I need to add usage parameter in filesys.conf too like this:
/,90
/run,99
If / usage greater than 90, /run usage greater than 99, write to log file.
How can I do this?
Let's create a bash associative array from the file filesys.conf on the form assoc[file system]=threshold. This way, we can loop over it and compare against the output of df -P.
Given the file:
$ cat filesys.conf
/,90
/dev,25
We store it in the array with (source: Bash reading from a file to an associative array):
declare -A assoc
while IFS=, read -r -a array;
do
assoc[${array[0]}]=${array[1]}
done < filesys.conf
So now the values are:
$ for k in "${!assoc[#]}"; do echo "${k} --> ${assoc[$k]}"; done
/dev --> 25
/ --> 90
Now it is just a matter of processing the output of df with, for example, awk:
mydate=$(date "+%d/%m/%Y")
for k in "${!assoc[#]}"
do
df -P "${k}" | awk 'NR==2 && $(NF-1)+0 > limit {print date, fs, $(NF-1)}' OFS=, fs="$k" date=mydate limit="${assoc[$k]}"
done >> disk.log
This pipes df to awk. There, we check the second line (the first one is the header, which apparently cannot be removed from the output (man df does not mention it)) and, in there, the value of the penultimate column. If its value is bigger than the given threshold, we print the desired output.
Note the trick of $(NF-1)+0 to cast the Use% column to int: the format 13% gets converted into 13.

How to properly use the grep command to grab and store integers?

I am currently building a bash script for class, and I am trying to use the grep command to grab the values from a simple calculator program and store them in the variables I assign, but I keep receiving a syntax error message when I try to run the script. Any advice on how to fix it? my script looks like this:
#!/bin/bash
addanwser=$(grep -o "num1 + num2" Lab9 -a 5 2)
echo "addanwser"
subanwser=$(grep -o "num1 - num2" Lab9 -s 10 15)
echo "subanwser"
multianwser=$(grep -o "num1 * num2" Lab9 -m 3 10)
echo "multianwser"
divanwser=$(grep -o "num1 / num2" Lab9 -d 100 4)
echo "divanwser"
modanwser=$(grep -o "num1 % num2" Lab9 -r 300 7)
echo "modawser"`
You want to grep the output of a command.
grep searches from either a file or standard input. So you can say either of these equivalent:
grep X file # 1. from a file
... things ... | grep X # 2. from stdin
grep X <<< "content" # 3. using here-strings
For this case, you want to use the last one, so that you execute the program and its output feeds grep directly:
grep <something> <<< "$(Lab9 -s 10 15)"
Which is the same as saying:
Lab9 -s 10 15 | grep <something>
So that grep will act on the output of your program. Since I don't know how Lab9 works, let's use a simple example with seq, that returns numbers from 5 to 15:
$ grep 5 <<< "$(seq 5 15)"
5
15
grep is usually used for finding matching lines of a text file. To actually grab a part of the matched line other tools such as awk are used.
Assuming the output looks like "num1 + num2 = 54" (i.e. fields are separated by space), this should do your job:
addanwser=$(Lab9 -a 5 2 | awk '{print $NF}')
echo "$addanwser"
Make sure you don't miss the '$' sign before addanwser when echo'ing it.
$NF selects the last field. You may select nth field using $n.

bash awk file compare

I have a config
[LogicalUnit1] UnitInquiry "NFSN00Y5IP51ZL" LUN0 /mnt/extent0 64MB
[LogicalUnit2] UnitInquiry "NFSN00N49CQL28" LUN0 /mnt/extent1 64MB
[LogicalUnit3] UnitInquiry "NFSNBRGQOCXK" LUN0 /mnt/extent4 10MB
[LogicalUnit4] UnitInquiry "NFSNE7IXADFJ" LUN0 /mnt/extent5 25MB
which is read via a bash script, using awk i parse the file and get variables
awk '/UnitInquiry/ {print $1, $3, $5, $6}' $ctld_config | while read a b c d ; do
if [ -f $a ]
then
ctladm create -b block -o file=$c -S $b -d $a
ctladm devlist -v > $lun_config
else
truncate -s $d $c ; ctladm create -b block -o file=$c -S $b -d $a
fi
this will initialize the luns properly on bootup, however if i add a lun then it will recreate them all again, how can i compare whats running, to whats configured and only reinitialize the ones not already live, there is a command to list the devices
ctladm devlist -v
LUN Backend Size (Blocks) BS Serial Number Device ID
0 block 131072 512 "NFSN00Y5IP51ZL [LogicalUnit1]
lun_type=0
num_threads=14
file=/mnt/extent0
1 block 131072 512 "NFSN00N49CQL28 [LogicalUnit2]
lun_type=0
num_threads=14
file=/mnt/extent1
2 block 20480 512 "NFSNBRGQOCXK" [LogicalUnit3]
lun_type=0
num_threads=14
file=/mnt/extent4
3 block 51200 512 "NFSNE7IXADFJ" [LogicalUnit4]
lun_type=0
num_threads=14
file=/mnt/extent5
Why not add the following after the then:
ctladm devlist -v | grep -q "$a" && continue
This will
run the command that show the currently active devices
check if the LogicalUnit name you want to register is already listed, and if yes...
skip the rest of the loop.
If $a (logical unit name) is not unique enough, you can also grep for another, more unique identifier, e.g. the serial number.

syntax error: operand expected (error token is ">= 75 ")

#!/bin/bash
CURRENT=$(df -h / | grep / | awk '{ print $4}')
THRESHOLD=75
if (( "$CURRENT" >= "$THRESHOLD" )); then
mail -s "CENTOS-6 localhost 10.10.1.238 Disk Space Alert" sss#abc.net << EOF
Your root partition remaining free space is critically low. Used: $CURRENT%
EOF
fi
I got the following error when i run the script, syntax error: operand expected (error token is ">= 75 ")
It's because CURRENT will contain a percent sign, so it won't be a valid operand for the comparison operation.
You can remove the last character like this :
CURRENT=${CURRENT%?};
Also make sure that df -h / | grep / | awk '{ print $4}' is correctly returning the usage ratio, on most systems you have to use print $5.
A couple of things:
you don't need grep at all, awk is quite capable of doing it's own regex stuff.
if you search for / in the df output, you'll probably get most lines as most mounts have a / somewhere in them. If you just want the root mountpoint, you can use <space>/$.
Check that 4 is the correct field number, on my box it's 5.
In any case, that field is of the form 55% which will not be considered numeric. You can use gsub to get rid of it.
With that in mind, the following snippet can be used to get the percentage:
df -h | awk '$0 ~ / \/$/ { gsub("%","",$5); print $5 }'
And, just as an aside, I'm not that big a fan of here-docs in shell scripts since it either (1) screws up my nicely indented files; or (2) makes me burn half an hour while I try to remember the various syntax options which will allow indented EOF strings :-)
I prefer something like:
(
echo Your root partition remaining free space is critically low: Used: ${CURRENT}%
) | mail -s "CENTOS-6 localhost 10.10.1.238 Disk Space Alert" sss#abc.net
Especially since that means I can put arbitrarily complex commands in the sub-shell to generate whatever info I want in the mail message (rather than just simple text substitutions).
So, bottom line, I'd be looking at something more like:
#!/usr/bin/env bash
# Config section.
LIMIT=75
# Code section.
CURR=$(df -h | awk '$0 ~ / \/$/ { gsub("%","",$5); print $5 }')
if [[ ${CURR} -ge ${LIMIT} ]] ; then
(
echo "Your root partition remaining free space is critically low: Used: ${CURR}%"
) | mail -s "CENTOS-6 localhost 10.10.1.238 Disk Space Alert" sss#abc.net
fi
Just try:
CURRENT=$(df -h |awk '{print $4}' |sort -n |tail -n1 |sed 's/%//g')
THRESHOLD=90
if [ $THRESHOLD -gt $CURRENT ]

Resources