Having difficulty defining conditions to call certain functions and error messages - bash

I'm writing a piece of code which will use data from a file that I've already made in order to work out the average value of the file, the minimum value, maximum value and then finally displaying all values at once.
I'm very new to unix so I'm trying to learn it but I just cant seem to crack where I need to go with my code in order for it to gain functionality.
I've got the basics of the code but I need to find a way to call the functions using the year, which is stored in a directory corresponding to that year which is making me think I'm going to have problems calling from the file as I'm using a sed function to only take line 4 of that file rather than the year.
I also need to figure out how to set error messages and status to the script if they have not stated (Year) (One of the 4 commands), the year doesnt correspond to one available in the tree and the keyword is invalid.
Any help or even pointers towards good material to learn these things would be great.
Here is my current code:
#!/bin/bash
#getalldata() {
#find . -name "ff_*" -exec sed -n '4p' {} \;
#}
#Defining where the population configuration file is which contains all the data
popconfile.txt=$HOME/testarea
#Function to find the average population of all of the files
averagePopulation()
{
total=0
list=$(cat popconfile.txt)
for var in "${list[#]}"
do
total=$((total + var))
done
average=$((total/$(wc -l popconfile.txt)))
echo "$average"
}
#Function to find the maximum population from all the files
maximumPopulation()
{
max=1
for in `cat popconfile.txt`
do
if [[ $1 > "$max" ]]; then
max=$1
echo "$max"
fi
done
}
#Function to find the minimum population from all the files
minimumPopulation()
{
min=1000000
for in `cat popconfile.txt`
do
if [[ $1 < "$min" ]]; then
max=$1
echo "$min"
fi
done
}
#Function to show all of the results in one place
function showAll()
{
echo "$min"
echo "$max"
echo "$average"
}
Thanks!

Assuming your popconfile.txt format is
cat popconfile.txt
150
10
45
1000
34
87
You might be able to simplify your code with :
for i in $(cat popconfile.txt);do
temp[$i]=$i
done
pop=(${temp[*]})
min=${pop[0]}
max=${pop[$((${#pop[*]}-1))]}
for ((j=0;j<${#pop[*]};j++));do
sum=$(($sum+${pop[$j]}))
done
average=$(($sum/${#pop[*]}))
echo "maximum="$max
echo "minimum="$min
echo "average="$average
Be aware though that the average here or in your code is calculated with integer mathematics, so you're loosing all decimals.

Related

How to exclude lines above a target

I have read multiple posts about how to exclude lines around a grep match, but none addresses it with finality, most find other ways to sort data, and that does not solve similar issues with different data.
i have a file with a recursive output, a command repeated over and over. i want to trim out the 0 results because it is the only constant value, the result hits are an unknown quantity.
the only unique string i can search by needs to have the 4 lines above it excluded no matter what the content of those lines are, and i have not found any post with info generic enough to fit.
this is a conceptual question, there has to be a simple solution, but if an example is needed:
Path/Path/Path> search
[results]
[results]
2 entries found
Path/Path/Path> search
[result]
1 entry found
Path/Path/Path> search
0 entry found
try this:
# Assumption: The data is in logile.txt
i=0
tac logfile.txt |\
while read -r line; do
if [[ "${line:0:7}" == "0 entry" ]]; then
i=0
continue
else
((i++))
[[ $i -le 4 ]] && continue
fi
echo "$line"
done | tac
output:
Path/Path/Path> search
[results]
[results]
2 entries found

How to optimize the following script

This script runs a query to get a list of dates and runs two other queries for these dates.
Then compares which one is the smaller of the numbers and multiplies it by 2.
Then writes to file and sums them.
Please suggest improvements. Also checks for 0 numbers.
#!/bin/bash
1>output.txt
today=$(date +"%Y%m%d")
FirstOfTheMonth=$(date -d "--$(($(date +%-d)-1)) day" `enter code here`+"%Y%m%d")
echo "XXXX activity report on daily and cumulative monthly `enter code here`basis "
#query that outputs dates to a file
SQL query > list
#for each date I run 2 queries
for i in `cat list`;do
a1=SQL query;
b1=SQL query;
# I compare to find out which one is the smaller number and `enter code here`multiply it by 2
buy=${a1#-}
sell=${b1#-}
echo "XXX report for $yesterday month = $i "
echo "Buy $buy"
echo "Sell $sell"
if [ "$buy" -lt "$sell" ];
then DayNumber=$[buy * 2];
else DayNumber=$[sell * 2];
fi;
#I write all the numbers to a file since I have to sum them
MonthNumber=`awk '{ sum += $1 } END { print sum }' `enter `enter code here`code here`DayNumber$i`
echo "Day Number $DayNumber"
echo "$DayNumber$i $MonthNumber$1 $yesterday" >> DayNumber$i
echo "Day Number since $FirstOfTheMonth $MonthNumber$1"
echo ---------------------------------------------------------------------------------------
done
/usr/bin/mail -s "XXXX report $today" xxx#xxxx.com < `enter code here`output.txt
You are most likely getting down votes, do to the size of the code, lack of clarity on what you want, not showing effort to get that result on your own, and very rudimentary mistakes in code readability that could be fixed with a web search for a tutorial. Google has a shell style guide that you should read.
If you are redirecting stdout to a file, you are probably running autonomously. Which means you should redirect stderr too.
exec &>output.txt
First of the month-- not sure why you don't just do this
FirstOfTheMonth=$(date +%Y%m1)
For Pete's sake, indent! This makes me want to beat scripters. Also, don't use i unless it's a very small loop. Use a variable that means something.
while read -rd' ' month; do
some commands
if [[ $buy -lt $sell ]]; then
do this thing here
fi
done
Bash isn't semi-colon terminated, you don't need one at the end of a line. Use the interan [[ ]] over the externam [ ]. Don't quote a variable (e.g. "$buy") to do a numeric comparison (e.g. -lt). Keep your then on the same line as if-- it servers no other purpose than to make it more readable.

Adding two decimal variables and assigning values in bash

we have been asked to parse a csv file and perform some operations based upon the data in the csv
I am trying to find the maximum of addition of two numbers which i get from the csv file
that is the last and second last numbers, which are decimals
Following is my code
#!/bin/bash
#this file was created on 09/03/2014
#Author = Shashank Pangam
OLDIFS=$IFS
IFS=","
maxTransport=0
while read year month hydro geo solar wind fuel1 biomassL biomassC totalRenew fuel2 biodieselT biomassT
do
while [ $year -eq 2012 ]
do
currentTransport=$(echo "$biodieselT+$biomassT" | bc)
echo $currentTransport
if (( $(echo "$currentTransport > $maxTransport" | bc -l)));
then
$maxTransport = $currentTransport
echo $maxTransport
fi
done
echo -e "Maximum amount of energy consumed by the Transportation sector for year 2012 : $maxTransport"
done < $1
and the following is my csv file
2012,January,2.614,0.356,0.006,0.021,114.362,14.128,1.308,66.74,196.539,199.536,81.791,
2012,February,2.286,0.333,0.007,0.017,107.388,13.952,1.304,61.277,183.921,186.564,81.545,
2012,March,0.356,0.009,0.02,108.268,15.588,1.404,63.444,188.705,191.318,87.827,11.187,
2012,April,,0.344,0.012,0.019,103.627,14.229,1.381,60.683,179.919,181.993,86.339,11.518,
2012,May,,0.356,0.012,0.01,109.644,13.789,1.473,63.611,188.517,190.913,92.087,12.09,
2012,June,,0.344,0.013,0.013,108.116,13.012,1.434,61.056,183.618,185.65,89.673,12.461,
2012,July,,0.356,0.017,0.008,112.426,14.035,1.403,58.057,185.921,187.61,87.707,10.464,
2012,August,0.356,0.016,0.008,113.64,14.01,1.513,60.011,189.174,190.999,94.592,11.14,
2012,September,1.513,0.344,0.015,0.01,110.84,13.435,1.324,56.047,181.647,183.528,82.814,
2012,October,1.83,0.356,0.012,0.02,111.544,15.597,1.462,57.365,185.969,188.186,91.42,
2012,November,2.022,0.344,0.01,0.014,111.808,15.594,1.326,56.793,185.521,187.911,82.919,
2012,December,1.77,0.356,0.007,0.022,116.416,15.873,1.368,58.741,192.398,194.552,85.526,
2013,January,3.021,0.357,0.007,0.018,114.601,15.309,1.334,57.31,188.553,191.956,83.415,
2013,February,3.285,0.322,0.012,0.023,102.499,13.658,1.246,52.05,169.452,173.094,77.914,
2013,March,0.357,0.016,0.025,111.594,14.538,1.419,59.096,186.646,189.884,88.713,11.938,
2013,April,,0.345,0.018,0.03,103.602,14.446,1.437,59.057,178.542,181.342,89.867,12.184,
2013,May,,0.357,0.02,0.032,108.113,14.452,1.497,62.606,186.668,190.117,93.634,13.166,
2013,June,,0.345,0.021,0.028,109.162,14.597,1.47,61.563,186.792,189.994,91.894,14.501,
2013,July,,0.357,0.018,0.024,119.154,15.018,1.45,62.037,197.659,201.027,90.689,14.523,
2013,August,0.357,0.022,0.02,113.177,15.014,1.44,60.682,190.313,192.949,90.065,13.28,
2013,September,2.185,0.345,0.021,0.026,106.912,14.367,1.411,58.901,181.591,184.168,88.254,
2013,October,2.171,0.357,0.02,0.029,109.123,15.158,1.483,64.509,190.273,192.849,92.748
The following is the error i get
./calculator.sh: line 16: 0: command not found
0
268.109
I don't understand why echo $currentTransport returns 0 while in the comparison it works and assigns value to maxTransport but throws the error for the same.
Thanks in advance.
Instead of this:
$maxTransport = $currentTransport
Try this:
maxTransport=$currentTransport
The $ in front of a variable gives its contents. By removing the $, the actual variable location of maxTransport is used instead as the destination for the contents of currentTransport.

Bash script that analyzes report files

I have the following bash script which I will use to analyze all report files in the current directory:
#!/bin/bash
# methods
analyzeStructuralErrors()
{
# do something with $1
}
# main
reportFiles=`find $PWD -name "*_report*.txt"`;
for f in $reportFiles
do
echo "Processing $f"
analyzeStructuralErrors $f
done
My report files are formatted as such:
Error Code for Issue X - Description Text - Number of errors.
col1_name,col2_name,col3_name,col4_name,col5_name,col6_name
1143-1-1411-247-1-72953-1
1143-2-1411-247-436-72953-1
2211-1-1888-204-442-22222-1
Error Code for Issue Y - Description Text - Number of errors.
col1_name,col2_name,col3_name,col4_name,col5_name,col6_name
Other data
.
.
.
I'm looking for a way to go through each file and aggregate the report data. In the above example, we have two unique issues of type X, which I would like to handle in analyzeStructural. Other types of issues can be ignored in this routine. Can anyone offer advice on how to do this? I want to read each line until I hit the next error basically, and put that data into some kind of data structure.
Below is a working awk implementation that uses it's pseudo multidimensional arrays. I've included sample output to show you how it looks. I took the liberty to add a 'Count' column to denote how many times a certain "Issue" was hit for a given Error Code
#!/bin/bash
awk '
/Error Code for Issue/ {
errCode[currCode=$5]=$5
}
/^ +[0-9-]+$/ {
split($0, tmpArr, "-")
error[errCode[currCode],tmpArr[1]]++
}
END {
for (code in errCode) {
printf("Error Code: %s\n", code)
for (item in error) {
split(item, subscr, SUBSEP)
if (subscr[1] == code) {
printf("\tIssue: %s\tCount: %s\n", subscr[2], error[item])
}
}
}
}
' *_report*.txt
Output
$ ./report.awk
Error Code: B
Issue: 1212 Count: 3
Error Code: X
Issue: 2211 Count: 1
Issue: 1143 Count: 2
Error Code: Y
Issue: 2961 Count: 1
Issue: 6666 Count: 1
Issue: 5555 Count: 2
Issue: 5911 Count: 1
Issue: 4949 Count: 1
Error Code: Z
Issue: 2222 Count: 1
Issue: 1111 Count: 1
Issue: 2323 Count: 2
Issue: 3333 Count: 1
Issue: 1212 Count: 1
As suggested by Dave Jarvis, awk will:
handle this better than bash
is fairly easy to learn
likely available wherever bash is available
I've never had to look farther than The AWK Manual.
It would make things easier if you used a consistent field separator for both the list of column names and the data. Perhaps you could do some pre-processing in a bash script using sed before feeding to awk. Anyway, take a look at multi-dimensional arrays and reading multiple lines in the manual.
Bash has one-dimensional arrays that are indexed by integers. Bash 4 adds associative arrays. That's it for data structures. AWK has one dimensional associative arrays and fakes its way through two dimensional arrays. If you need some kind of data structure more advanced than that, you'll need to use Python, for example, or some other language.
That said, here's a rough outline of how you might parse the data you've shown.
#!/bin/bash
# methods
analyzeStructuralErrors()
{
local f=$1
local Xpat="Error Code for Issue X"
local notXpat="Error Code for Issue [^X]"
while read -r line
do
if [[ $line =~ $Xpat ]]
then
flag=true
elif [[ $line =~ $notXpat ]]
then
flag=false
elif $flag && [[ $line =~ , ]]
then
# columns could be overwritten if there are more than one X section
IFS=, read -ra columns <<< "$line"
elif $flag && [[ $line =~ - ]]
then
issues+=(line)
else
echo "unrecognized data line"
echo "$line"
fi
done
for issue in ${issues[#]}
do
IFS=- read -ra array <<< "$line"
# do something with ${array[0]}, ${array[1]}, etc.
# or iterate
for field in ${array[#]}
do
# do something with $field
done
done
}
# main
find . -name "*_report*.txt" | while read -r f
do
echo "Processing $f"
analyzeStructuralErrors "$f"
done

Convert floating point variable to integer?

The shell script shown below will show a warning if the page takes more than 6 seconds to load. The problem is that the myduration variable is not an integer. How do I convert it to integer?
myduration=$(curl http://192.168.50.1/mantisbt/view.php?id=1 -w %{time_total}) > /dev/null ; \
[[ $myduration -gt 1 ]] && echo "`date +'%y%m%d%H%M%S'
It took more than 6 seconds to load the page http://192.168.50.1/mantisbt/view.php?id=1
Assuming $myduration is a decimal or integer
$ myduration=6.5
$ myduration=$( printf "%.0f" $myduration )
$ echo $myduration
6
You can do this:
float=1.23
int=${float%.*}
I am using this on bash.
It's not entirely clear, but I think you're asking how to convert a floating-point value (myduration) to an integer in bash. Something like this may help you, depending on which way you want to round your number.
#!/bin/bash
floor_val=
ceil_val=
function floor() {
float_in=$1
floor_val=${float_in/.*}
}
function ceiling() {
float_in=$1
ceil_val=${float_in/.*}
ceil_val=$((ceil_val+1))
}
float_val=$1
echo Passed in: $float_val
floor $float_val
ceiling $float_val
echo Result of floor: $floor_val
echo Result of ceiling: $ceil_val
Example usage:
$ ./int.sh 12.345
Passed in: 12.345
Result of floor: 12
Result of ceiling: 13
Eliminate page contents from the variable:
When I tried your command, myduration contained the HTML contents of the page at the URL I used in my test plus the time value. By adding -s to suppress the progress bar and adding -o /dev/null to the options for curl, I was able to remove the redirect to /dev/null and have only the time saved in myduration.
Since the value of myduration is likely to be short, you can use the technique ire_and_curses shows which will often yield zero as its result which would be less than the 1 you are testing for (note that your log message says "6 seconds", though).
Finer resolution:
If you'd like to have a finer resolution test, you can multiply myduration by 1000 using a technique like this:
mult1000 () {
local floor=${1%.*}
[[ $floor = "0" ]] && floor=''
local frac='0000'
[[ $floor != $1 ]] && frac=${1#*.}$frac
echo ${floor}${frac:0:3}
}
Edit: This version of mult1000 properly handles values such as "0.234", "1", "2.", "3.5"
and "6.789". For values with more than three decimal places, the extra digits are truncated without rounding regardless of the value ("1.1119" becomes "1.111").
Your script with the changes I mentioned above and using mult1000 (with my own example time):
myduration=$(curl -s -o /dev/null http://192.168.50.1/mantisbt/view.php?id=1 -w %{time_total}); [[ $(mult1000 $myduration) -gt 3500 ]] && echo "`date +'%y%m%d%H%M%S'` took more than 3.5 seconds to load the page http://192.168.50.1/mantisbt/view.php?id=1 " >> /home/shantanu/speed_report.txt
Here it is broken into multiple lines (and simplified) to make it more readable here in this answer:
myduration=$(curl -s -o /dev/null http://example.com -w %{time_total})
[[ $(mult1000 $myduration) -gt 3500 ]] &&
echo "It took more than 3.5 seconds to load thttp://example.com" >> report.txt

Resources