Create bins with awk histogram-like - bash

Question

Here's my input file :
1.37987
1.21448
0.624999
1.28966
1.77084
1.088
1.41667
I would like to create bins of a size of my choice to get histogram-like output, e.g. something like this for 0.1 bins, starting from 0 :
0 0.1 0
...
0.5 0.6 0
0.6 0.7 1
...
1.0 1.1 1
1.1 1.2 0
1.2 1.3 2
1.3 1.4 1
...
My file is too big for R, so I'm looking for an awk solution (also open to anything else that I can understand, as I'm still a Linux beginner).
This was sort of already answered in this post : awk histogram in buckets but the solution is not working for me.

Answer 1

This should be very close if not exactly right. Consider it a starting point at least and verify/figure out the math yourself (in particular decide/verify which bucket(s) an exact boundary match like 0.2 should go into - 0.1 to 0.2 and/or 0.2 to 0.3?):
$ cat tst.awk
BEGIN { delta = (delta == "" ? 0.1 : delta) }
{
bucketNr = int(($0+delta) / delta)
cnt[bucketNr]++
numBuckets = (numBuckets > bucketNr ? numBuckets : bucketNr)
}
END {
for (bucketNr=1; bucketNr<=numBuckets; bucketNr++) {
end = beg + delta
printf "%0.1f %0.1f %d\n", beg, end, cnt[bucketNr]
beg = end
}
}
$ awk -f tst.awk file
0.0 0.1 0
0.1 0.2 0
0.2 0.3 0
0.3 0.4 0
0.4 0.5 0
0.5 0.6 0
0.6 0.7 1
0.7 0.8 0
0.8 0.9 0
0.9 1.0 0
1.0 1.1 1
1.1 1.2 0
1.2 1.3 2
1.3 1.4 1
1.4 1.5 1
1.5 1.6 0
1.6 1.7 0
1.7 1.8 1
Note that you can assign the bucket delta size on the command line, 0.1 is just the default value:
$ awk -v delta='0.3' -f tst.awk file
0.0 0.3 0
0.3 0.6 0
0.6 0.9 1
0.9 1.2 1
1.2 1.5 4
1.5 1.8 1
$ awk -v delta='0.5' -f tst.awk file
0.0 0.5 0
0.5 1.0 1
1.0 1.5 5
1.5 2.0 1

Answer 2

This is also possible :
awk -v size=0.1
'{ b=int($1/size); a[b]++; bmax=b>bmax?b:bmax; bmin=b<bmin?b:bmin }
END { for(i=bmin;i<=bmax;++i) print i*size,(i+1)*size,a[i] }' <file>
It essentially does the same as the solution of EdMorton, but starts printing buckets from the minimum value which is default 0. It essentially takes negative numbers into account.

Answer 3

Here is my stab at solving this with Awk.
To run: awk -f belowscript.awk inputfile
BEGIN {
PROCINFO["sorted_in"]="#ind_num_asc";
delta = (delta == "") ? 0.1 : delta;
};
/^-?([0-9][0-9]*|[0-9]*(\.[0-9][0-9]*))/ {
# Special case the [-delta - 0] case so it doesn't bin in the [0-delta] bin
fractBin=$1/delta
if (fractBin < 0 && int(fractBin) == fractBin)
fractBin = fractBin+1
prefix = (fractBin <= 0 && int(fractBin) == 0) ? "-" : ""
bins[prefix int(fractBin)]++
}
END {
for (var in bins)
{
srange = sprintf("%0.2f",delta * ((var >= 0) ? var : var-1))
erange = sprintf("%0.2f",delta * ((var >= 0) ? var+1 : var))
print srange " " erange " " bins[var]
}
}
Some notes:
I added support for providing the bin size on the command line like Ed Morton did.
It only prints the bins that contain something
Which bin an exact match goes in - the smaller or the larger bin naturally with this approach negated when going negative, and required tweaking to make it consistent.
the 0 boundary needed special casing for those numbers in the first negative bin, since there is no such number as -0. Awk's associative arrays use strings for keys, so "-0" was possible, and with #ind_num_asc sort order for the for loop, seems to sort the -0 properly - though this may not be portable.

Answer 4

Another solution with Python
# draw histogram in command line with Python
#
# usage: $ cat datafile.txt | python this_script.py [nbins] [nscale]
# The input should be one column of numbers to be piped in.
#
# forked from https://gist.github.com/bgbg
from __future__ import print_function
import sys
import numpy as np
def asciihist(it, bins=10, minmax=None, str_tag='',
scale_output=30, generate_only=False, print_function=print):
"""Create an ASCII histogram from an interable of numbers.
Author: Boris Gorelik boris#gorelik.net. based on http://econpy.googlecode.com/svn/trunk/pytrix/pytrix.py
License: MIT
"""
ret = []
itarray = np.asanyarray(it)
if minmax == 'auto':
minmax = np.percentile(it, [5, 95])
if minmax[0] == minmax[1]:
# for very ugly distributions
minmax = None
if minmax is not None:
# discard values that are outside minmax range
mn = minmax[0]
mx = minmax[1]
itarray = itarray[itarray >= mn]
itarray = itarray[itarray <= mx]
if itarray.size:
total = len(itarray)
counts, cutoffs = np.histogram(itarray, bins=bins)
cutoffs = cutoffs[1:]
if str_tag:
str_tag = '%s ' % str_tag
else:
str_tag = ''
if scale_output is not None:
scaled_counts = counts.astype(float) / counts.sum() * scale_output
else:
scaled_counts = counts
if minmax is not None:
ret.append('Trimmed to range (%s - %s)' % (str(minmax[0]), str(minmax[1])))
for cutoff, original_count, scaled_count in zip(cutoffs, counts, scaled_counts):
ret.append("{:s}{:>8.2f} |{:<7,d} | {:s}".format(
str_tag,
cutoff,
original_count,
"*" * int(scaled_count))
)
ret.append(
"{:s}{:s} |{:s} | {:s}".format(
str_tag,
'-' * 8,
'-' * 7,
'-' * 7
)
)
ret.append(
"{:s}{:>8s} |{:<7,d}".format(
str_tag,
'N=',
total
)
)
else:
ret = []
if not generate_only:
for line in ret:
print_function(line)
ret = '\n'.join(ret)
return ret
if __name__ == '__main__':
nbins=30
if len(sys.argv) >= 2:
nbins = int(sys.argv[1])
nscale=400
if len(sys.argv) == 3:
nscale = int(sys.argv[2])
dataIn =[]
for line in sys.stdin:
if line.strip() != '':
dataIn.append( float(line))
asciihist(dataIn, bins=nbins, scale_output=nscale, minmax=None, str_tag='BIN');