Bash math - Dividing a bunch of rows using for statement - bash

Example file:
25 Firstname1 Lastname1 domain1.com planname #1.00 USD Monthly Active 04/24/2016 Edit
1068 Firstname2 Lastname2 domain2.com planname #7.95 USD Annually Active 05/09/2016 Edit
3888 Firstname3 Lastname3 domain3.com planname #19.95 USD Biennially Active 05/04/2016 Edit
I am extracting just the price and billing cycle and am converting the billing cycles into numerical value this way I can divide the price by the billing cycle to get a cost per month.
When using the for statement, its adding line breaks which is breaking the math.
Code:
for i in `cat asd | cut -d "#" -f 2 | awk '{print $1, $3}' | sed 's/Monthly/\/ 1/g' | sed 's/Annually/\/ 12/g' | sed 's/Biennially/\/ 24/g' |grep -Ev 0.00` ; do echo $i | bc -l' ; done
I would prefer to be able to get 1 answer meaning all the rows get divided up then added together to get one final answer.

All those calls to cat, cut, awk, sed, grep and bc - what a waste.
This is a mis-named post, because you are not using Bash to do any calculations. The reason is that bash, unlike korn shell (ksh), does not support floating point. So you fall back to utilities like bc. Hold on though, awk supports floating point as well.
awk is a programming language in its own right. This just uses one instance of awk. I have embedded it inside a bash script because you are probably doing other stuff, but with a little adjustment it could be stand-alone with #!/bin/awk at the top:
infile='asd'
# -f - means "read the program from stdin"
# << '_END_' is a here document. Redirect stdin from here to the label _END_
awk -f - "$infile" << '_END_'
BEGIN {
# an associative array for the billing cycles
cycles["Monthly"] = 1
cycles["Annually"] = 12
cycles["Biennially"] = 24
}
{
sub(/#/,"",$6) # Remove the # from the amount
total += $6/cycles[$8] # divide amount by the billing cycle, add to total
}
END { print total }
_END_
Don't you think this is simpler to understand and maintain? It's also more efficient. This awk script is probably a good exercise for an awk 101 training course.

You could do something like this: (If you are totally set on a single line)
cat asd | cut -d "#" -f 2 | awk '{print $1, $3}' | sed 's/Monthly/\/ 1/g' | sed 's/Annually/\/ 12/g' | sed 's/Biennially/\/ 24/g' | grep -Ev 0.00 | while IFS= read -r line; do echo "$line" | bc -l; done | tr '\n' '+' | sed 's/+$/\n/' | bc -l
But this would be way more clear:
tmp=$(mktemp)
cat asd | cut -d "#" -f 2 | awk '{print $1, $3}' | sed 's/Monthly/\/ 1/g' | sed 's/Annually/\/ 12/g' | sed 's/Biennially/\/ 24/g' | grep -Ev 0.00 > $tmp
tmp2=$(mktemp)
cat $tmp | while IFS= read -r line; do
echo "$line" | bc -l >> $tmp2
done
# Actual output
cat $tmp2 | tr '\n' '+' | sed 's/+$/\n/' | bc -l
rm $tmp $tmp2

Related

Split pipe into two and paste the results together?

I want to pipe the output of the command into two commands and paste the results together. I found this answer and similar ones suggesting using tee but I'm not sure how to make it work as I'd like it to.
My problem (simplified):
Say that I have a myfile.txt with keys and values, e.g.
key1 /path/to/file1
key2 /path/to/file2
What I am doing right now is
paste \
<( cat myfile.txt | cut -f1 ) \
<( cat myfile.txt | cut -f2 | xargs wc -l )
and it produces
key1 23
key2 42
The problem is that cat myfile.txt is repeated here (in the real problem it's a heavier operation). Instead, I'd like to do something like
cat myfile.txt | tee \
<( cut -f1 ) \
<( cut -f2 | xargs wc -l ) \
| paste
But it doesn't produce the expected output. Is it possible to do something similar to the above with pipes and standard command-line tools?
This doesn't answer your question about pipes, but you can use AWK to solve your problem:
$ printf %s\\n 1 2 3 > file1.txt
$ printf %s\\n 1 2 3 4 5 > file2.txt
$ cat > myfile.txt <<EOF
key1 file1.txt
key2 file2.txt
EOF
$ cat myfile.txt | awk '{ ("wc -l " $2) | getline size; sub(/ .+$/,"",size); print $1, size }'
key1 3
key2 5
On each line we first we run wc -l $2 and save the result into a variable. Not sure about yours, but on my system wc -l includes the filename in the output, so we strip it with sub() to match your example output. And finally, we print the $1 field (key) and the size we got from wc -l command.
Also, can be done with shell, now that I think about it:
cat myfile.txt | while read -r key value; do
printf '%s %s\n' "$key" "$(wc -l "$value" | cut -d' ' -f1)"
done
Or more generally, by piping to two commands and using paste, therefore answering the question:
cat myfile.txt | while read -r line; do
printf %s "$line" | cut -f1
printf %s "$line" | cut -f2 | xargs wc -l | cut -d' ' -f1
done | paste - -
P.S. The use of cat here is useless, I know. But it's just a placeholder for the real command.

Get avg of cpu temp's as a one line script

I need to average the temperatures of the four cpu cores on my system. I am obtaining the temperatures of the individual cores using the command:
sysctl -a | awk '/temperature/ {print $2;}'
This spits out the following output:
53.0C
53.0C
52.0C
52.0C
I then pass this to sed and tr and with some script-fu I ended up with the following-one liner:
echo `sysctl -a | awk '/temperature/ {print $2;}' | sed s/C// | tr '\n' '+' | sed 's/\(.*\)+/\1/'` | bc`
which then results in:
210
I now simply need to divide 210/4 to get my average but am stumped on how to achieve this as an extension to the one-liner that I have already brewed up. And due to some other constraints, I need to keep this as a one-liner.
I am sure there's a simpler way to achieve what I am after, any pointers are appreciated!
With awk:
sysctl -a | awk -F '[ C]' '/temperature/{sum+=$2} END{print sum/NF}'
Output:
52.5
See: 8 Powerful Awk Built-in Variables – FS, OFS, RS, ORS, NR, NF, FILENAME, FNR
You can add parenthesis and division by 4 around your expression with:
{ echo -n "("; tr '\n' '+'; echo -n")/4"; }
The final result is:
echo `sysctl -a | awk '/temperature/ {print $2;}' | sed s/C// | { echo -n "("; tr '\n' '+'; echo -n")/4"; } | sed 's/\(.*\)+/\1/'` | bc`

count all the lines in all folders in bash [duplicate]

wc -l file.txt
outputs number of lines and file name.
I need just the number itself (not the file name).
I can do this
wc -l file.txt | awk '{print $1}'
But maybe there is a better way?
Try this way:
wc -l < file.txt
cat file.txt | wc -l
According to the man page (for the BSD version, I don't have a GNU version to check):
If no files are specified, the standard input is used and no file
name is
displayed. The prompt will accept input until receiving EOF, or [^D] in
most environments.
To do this without the leading space, why not:
wc -l < file.txt | bc
Comparison of Techniques
I had a similar issue attempting to get a character count without the leading whitespace provided by wc, which led me to this page. After trying out the answers here, the following are the results from my personal testing on Mac (BSD Bash). Again, this is for character count; for line count you'd do wc -l. echo -n omits the trailing line break.
FOO="bar"
echo -n "$FOO" | wc -c # " 3" (x)
echo -n "$FOO" | wc -c | bc # "3" (√)
echo -n "$FOO" | wc -c | tr -d ' ' # "3" (√)
echo -n "$FOO" | wc -c | awk '{print $1}' # "3" (√)
echo -n "$FOO" | wc -c | cut -d ' ' -f1 # "" for -f < 8 (x)
echo -n "$FOO" | wc -c | cut -d ' ' -f8 # "3" (√)
echo -n "$FOO" | wc -c | perl -pe 's/^\s+//' # "3" (√)
echo -n "$FOO" | wc -c | grep -ch '^' # "1" (x)
echo $( printf '%s' "$FOO" | wc -c ) # "3" (√)
I wouldn't rely on the cut -f* method in general since it requires that you know the exact number of leading spaces that any given output may have. And the grep one works for counting lines, but not characters.
bc is the most concise, and awk and perl seem a bit overkill, but they should all be relatively fast and portable enough.
Also note that some of these can be adapted to trim surrounding whitespace from general strings, as well (along with echo `echo $FOO`, another neat trick).
How about
wc -l file.txt | cut -d' ' -f1
i.e. pipe the output of wc into cut (where delimiters are spaces and pick just the first field)
How about
grep -ch "^" file.txt
Obviously, there are a lot of solutions to this.
Here is another one though:
wc -l somefile | tr -d "[:alpha:][:blank:][:punct:]"
This only outputs the number of lines, but the trailing newline character (\n) is present, if you don't want that either, replace [:blank:] with [:space:].
Another way to strip the leading zeros without invoking an external command is to use Arithmetic expansion $((exp))
echo $(($(wc -l < file.txt)))
Best way would be first of all find all files in directory then use AWK NR (Number of Records Variable)
below is the command :
find <directory path> -type f | awk 'END{print NR}'
example : - find /tmp/ -type f | awk 'END{print NR}'
This works for me using the normal wc -l and sed to strip any char what is not a number.
wc -l big_file.log | sed -E "s/([a-z\-\_\.]|[[:space:]]*)//g"
# 9249133

sort fields within a line

input:
87 6,1,9,13
3 9,4,14,35,38,13
31 3,1,6,5
(i.e. a tab-delimited column where the second field is a comma-delimited list of unordered integers.)
desired output:
87 1,6,9,13
3 4,9,13,14,35,38
31 1,3,5,6
Goal:
for each line separately, sort the comma-separated list appearing in the second field. i.e. sort the 2nd column within for each line separately.
Note: the rows should not be re-ordered.
What I've tried:
sort - Since the order of the rows should not change, then sort is simply not applicable.
awk - since the greater file is tab-delimited, not comma-delimited, it cannot parse the second column as multiple "sub-fields"
There might be a perl way? I know nothing about perl though...
It can be done by simple perl oneliner:
perl -F'/\t/' -alne'$s=join",",sort{$a<=>$b}split",",$F[1];print"$F[0]\t$s"'
and shell (bash) one as well:
while read a b;do echo -e "$a\t$(echo $b|tr , '\n'|sort -n|tr '\n' ,|sed 's/,$//')"; done
while read LINE; do
echo -e "$(echo $LINE | awk '{print $1}')\t$(echo $LINE | awk '{print $2}' | tr ',' '\n' | sort -n | paste -s -d,)";
done < input
Obviously a lot going on here so here we go:
input contains your input
$(echo $LINE | awk '{print $1}') prints the first field, pretty straightforward
$(echo $LINE | awk '{print $2}' | tr ',' '\n' | sort -n | paste -s -d,) prints the second field, but breaks it down into lines by replacing the commas by newlines (tr ',' '\n'), then sort numerically, then assemble the lines back to comma-delimited values (paste -s -d,).
$ cat input
87 6,1,9,13
3 9,4,14,35,38,13
31 3,1,6,5
$ while read LINE; do echo -e "$(echo $LINE | awk '{print $1}')\t$(echo $LINE | awk '{print $2}' | tr ',' '\n' | sort -n | paste -s -d,)"; done < input
87 1,6,9,13
3 4,9,13,14,35,38
31 1,3,5,6
Another way:
echo happybirthday|awk '{split($0,A);asort(A); for (i=1;i<length(A);i++) {print A[i]}}' FS=""|tr -d '\n';echo aabdhhipprty
I didn't know how to get back to this page after recovering login info, so am posting as a guest.

How to get "wc -l" to print just the number of lines without file name?

wc -l file.txt
outputs number of lines and file name.
I need just the number itself (not the file name).
I can do this
wc -l file.txt | awk '{print $1}'
But maybe there is a better way?
Try this way:
wc -l < file.txt
cat file.txt | wc -l
According to the man page (for the BSD version, I don't have a GNU version to check):
If no files are specified, the standard input is used and no file
name is
displayed. The prompt will accept input until receiving EOF, or [^D] in
most environments.
To do this without the leading space, why not:
wc -l < file.txt | bc
Comparison of Techniques
I had a similar issue attempting to get a character count without the leading whitespace provided by wc, which led me to this page. After trying out the answers here, the following are the results from my personal testing on Mac (BSD Bash). Again, this is for character count; for line count you'd do wc -l. echo -n omits the trailing line break.
FOO="bar"
echo -n "$FOO" | wc -c # " 3" (x)
echo -n "$FOO" | wc -c | bc # "3" (√)
echo -n "$FOO" | wc -c | tr -d ' ' # "3" (√)
echo -n "$FOO" | wc -c | awk '{print $1}' # "3" (√)
echo -n "$FOO" | wc -c | cut -d ' ' -f1 # "" for -f < 8 (x)
echo -n "$FOO" | wc -c | cut -d ' ' -f8 # "3" (√)
echo -n "$FOO" | wc -c | perl -pe 's/^\s+//' # "3" (√)
echo -n "$FOO" | wc -c | grep -ch '^' # "1" (x)
echo $( printf '%s' "$FOO" | wc -c ) # "3" (√)
I wouldn't rely on the cut -f* method in general since it requires that you know the exact number of leading spaces that any given output may have. And the grep one works for counting lines, but not characters.
bc is the most concise, and awk and perl seem a bit overkill, but they should all be relatively fast and portable enough.
Also note that some of these can be adapted to trim surrounding whitespace from general strings, as well (along with echo `echo $FOO`, another neat trick).
How about
wc -l file.txt | cut -d' ' -f1
i.e. pipe the output of wc into cut (where delimiters are spaces and pick just the first field)
How about
grep -ch "^" file.txt
Obviously, there are a lot of solutions to this.
Here is another one though:
wc -l somefile | tr -d "[:alpha:][:blank:][:punct:]"
This only outputs the number of lines, but the trailing newline character (\n) is present, if you don't want that either, replace [:blank:] with [:space:].
Another way to strip the leading zeros without invoking an external command is to use Arithmetic expansion $((exp))
echo $(($(wc -l < file.txt)))
Best way would be first of all find all files in directory then use AWK NR (Number of Records Variable)
below is the command :
find <directory path> -type f | awk 'END{print NR}'
example : - find /tmp/ -type f | awk 'END{print NR}'
This works for me using the normal wc -l and sed to strip any char what is not a number.
wc -l big_file.log | sed -E "s/([a-z\-\_\.]|[[:space:]]*)//g"
# 9249133

Resources