Bash Script to process irregular text, count occurrences, cut at a threshold - bash

Question

I have a large sample of text that is pretty irregular and I'd like to tokenize it to single words, and count occurrences of each words, and have an output where occurrence > threshold_value
if [ $# -ne 3 ]; then
echo 'Usage <file> <output_file> <threshold>'
exit 1
fi
clean_and_rank () {
tr -dc [:graph:][:cntrl:][:space:] < $1 \
| tr -d [:punct:] \
| tr -s ' ' \
| tr ' ' '\n' \
| tr '[A-Z]' '[a-z]' \
| grep -v '^$' \
| sort \
| uniq -c \
| sort -nr
}
cut_below_threshold () {
$THRESHOLD=$1
awk '$1 > '$THRESHOLD' { print $1, $2 }'
}
clean_and_rank $1 \
| cut_below_threshold $3
| sort -nr > $2
but for some reason I'm running into trouble with the cut_below_threshold() function --
Also once I get this done, I want to be able to compare it to another sample (my data is 2 samples of several rows of labeled text snippets and I want to score words independently for prevalence in sample A/sample B)
Is there a better way to go about this? Ultimately, I'm looking for insights along the lines of "$WORD is in sample 1 1000 times, out of 100000 total words, it is in sample 2 100 times out of 10000 words"

Answer 1

I assume you are able to get statistics for two text files in the following format:
$ cat a.txt
5 word1
3 word2
1 word3
$ cat b.txt
4 word1
3 word2
1 word4
Then this script does the job of comparison:
#!/bin/sh
# the 1st argument passed to the script, the 1st file to compare (statistics for sample A)
STATA=$1
# the 2nd argument -- the 2nd file (statistics for sample B)
STATB=$2
# concatenate both files and pipe it to the next command
cat ${STATA} ${STATB} |
# call awk; -v is awk option to set a variable
# n1=$() variable n1 gets its value from the output of the command in ()
# wc -l <file counts number of lines in the file
# ' starts awk script
awk -v n1=$(wc -l <${STATA}) '
# (){} means when condition in () is true, execute statement in {}
# NR is number of records processed thus far (usually this is number of lines)
# (NR <= n1) essentially means 'reading statistics file for sample A'
# {1; 2} two statements
# wa += $1 add value of the first field to the wa variable
# each line is splitted by a field separator (space or tab by default) into several fields:
# $1 is the 1st field, $2 is the 2nd, $NF is the last one, $0 is a whole line
# $1 in this case is number of occurrences of a word
# awk variables have zero default value; no need to specify them explicitly
# cnta[] is an associative array -- index is a string (the word in this case)
# $2 in this case is the word
(NR <= n1){wa += $1; cnta[$2] = $1}
# the same for statistics for sample B
(NR > n1){wb += $1; cntb[$2] = $1}
# END{} to execute statements after there's no input left
END {
print "nof words in sample A = " wa;
print "nof words in sample B = " wb;
# standard printf to output a table header
printf "%-15s %5s %8s %5s %8s\n", "word", "cntA", "freqA", "cntB", "freqB";
# iterate over each element (the word) in the count array A
for (w in cnta){
# check that the word is present in the count array B
if (cntb[w] > 0) {
# output statistics in a table form
printf "%-15s %5d %8.6f %5d %8.6f\n", w, cnta[w], cnta[w] / wa, cntb[w], cntb[w]/wb
}
}
}
'
Test run:
$ ./compare.sh a.txt b.txt
nof words in sample A = 9
nof words in sample B = 8
word cntA freqA cntB freqB
word1 5 0.555556 4 0.500000
word2 3 0.333333 3 0.375000

Answer 2

Let bash do most of your work with associative arrays. This is not a rigorous example, that is left as an exercise for you:
declare -A ct
exec 3< file
while IFS= read -u3 line ; do
set -- $line
for tkn ; do
cct=${ct[$tkn]}
ct[$tkn]=$(( ${cct:-0} + 1 ))
done
done
for tkn in ${!ct[*]}
do echo $tkn ${ct[$tkn]} ; done