I have a text file like below
11:00AM JOHN STAMOS 1983-08-07 I like Pizza Hut
12:00AM JACK SPARROW PIRATE 1886-09-07 I like Pizza Hut and DOminoz
11:00AM SANTA 1986-04-01 I like cold beer
How do I sort the above file on the date column? The problem I am facing is due to the variable length name column. Some people have first middle last name where as some have only first name and so on.
sed 's/\([0-9]\{4\}\(-[0-9]\{2\}\)\{2\}\)/|\1/' | sort -t '|' -k 2| sed s/'|'//
What you need to do is copy the date to the front and then sort which by default will use the whole line as the sort-key. Then remove the date again.
I used sed to pick out everything up to the (last) date which I located by its nnnn-nn-nn format, and copy the date to the front.
After the sort, just use sed (or cut -c11- would be easier) to delete the date from the front again.
This works in linux:
sed 's/^\(.* \([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] \)\)/\2\1/' |
sort |
sed 's/^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] //'
Giving:
12:00AM JACK SPARROW PIRATE 1886-09-07 I like Pizza Hut and DOminoz
11:00AM JOHN STAMOS 1983-08-07 I like Pizza Hut
11:00AM SANTA 1986-04-01 I like cold beer
This works for your data but could easily get pretty awkward if your data changes (for example you have multiple dates in a line).
cat file.txt | python -c 'import re, sys; print "".join(sorted(sys.stdin, key=lambda x:re.findall("\d{4}-\d{2}-\d{2}",x)))'
Pure Bash:
declare -a array
declare -a order
IFS=$'\n'
array=( $(cat "$infile") )
index=0
for line in "${array[#]}"; do
[[ "$line" =~ ([[:digit:]]+)-([[:digit:]]+)-([[:digit:]]+) ]]
key="${BASH_REMATCH[1]}${BASH_REMATCH[2]}${BASH_REMATCH[3]}"
if [ -z "${order[key]}" ] ; then
order[key]="$index"
else
order[key]="${order[key]} $index"
fi
((index++))
done < "$infile"
IFS=' '
for key in ${order[*]}; do
printf "%s\n" "${array[key]}"
done
Generates indices from dates and uses them as a sorted list.
Related
I have a .csv file which has dates and the answer about enjoyable or not:
2019-04-1,enjoyable
2019-04-2,unenjoyable
2019-04-3,unenjoyable
2019-04-4,enjoyable
2019-04-5,unenjoyable
2019-04-6,unenjoyable
2019-04-7,enjoyable
2019-04-8,unenjoyable
2019-04-9,unenjoyable
2019-04-10,enjoyable
2019-04-11,enjoyable
2019-04-12,enjoyable
2019-04-13,unenjoyable
2019-04-14,enjoyable
2019-04-15,unenjoyable
2019-04-16,unenjoyable
2019-04-17,unenjoyable
2019-04-18,enjoyable
2019-04-19,unenjoyable
2019-04-20,unenjoyable
2019-04-21,unenjoyable
2019-04-22,unenjoyable
2019-04-23,unenjoyable
2019-04-24,unenjoyable
2019-04-25,unenjoyable
2019-04-26,unenjoyable
What I want to do is to print the day of the week in the third column seperate by ',' like this:
2019-04-1,enjoyable,2
2019-04-2,unenjoyable,3
I tried:
dates=$(awk '{FS=","}{print $1,$2}' weather_stat.csv')
weeks=$(
for vars in $dates[first_row]
do
echo $(date -j -f '%Y-%m-%d' $vars "+%w")
done
)
merge($dates,$weeks)
The first part of the code works without any problem, but in the second part, I am confused about how to get the data in the first row (so I use dates[first_row] to mean the first row in dates variable) from the variable "dates" so we can apply 'date' method on it
And for the third part, I want to merge these two tables together. I found the 'join' function but it seem to work on two files instead of two variables(I don't want to have any new files during the process)
Could anyone tells me how to get the rows in a variable instead of a file in shell and the way to merge two table-like variables?
As you're learning shell scripting, here's some code to study:
to read your csv file, and get the weekday number for each date in the file:
while IFS=, read -r date rest; do echo "$date,$(date -d "$date" +%w)"; done < file.csv
to join the output of that command with your file:
weekdays=$(while IFS=, read -r date rest; do echo "$date,$(date -d "$date" +%w)"; done < file.csv)
join -t, file.csv <(echo "$weekdays")
or, without needing to store the result in an intermediate variable
join -t, file.csv <(
while IFS=, read -r date rest; do echo "$date,$(date -d "$date" +%w)"; done < file.csv
)
The newlines within the <() are not necessary, but useful for maintainable code.
However, you can see that this is less efficient because you have to process the file twice. With awk you only have to read through the file once.
With GNU awk:
awk' BEGIN{FS=OFS=","}
{ split($1,a,"-")
t=sprintf("%0.4d %0.2d %0.2d 00 00 00",a[1],a[2],a[3]);
print $0,strftime("%w",mktime(t))
}' file.csv
With only your Bourne shell, so less efficient than awk if you have a lot of lines in your CSV file:
while IFS=, read date enjoy; do
date -d "$date" +"$date,$enjoy,%w"
done < your.csv
I want to stick all the variables into one variable
A=('blah')
AA=('blah2')
AAA=('blah3')
AAB=('blah4')
AAC=('blah5')
#^^lets pretend theres 100 more of these ^^
#Variable composition
#after AAA, is AAB then AAC then AAD etc etc, does that 100 times
I want them all placed into this MASTER variable
#MASTER=${A}${AA}${AAA} (<-- insert AAB, AAC and 100 more variables here)
I obviously don't want to type 100 variables in this expression because there's probably an easier way to do this. Plus I'm gonna be doing more of these therefore I need it automated.
I'm relatively new to sed, awk, is there a way to append those 100 variables into the master variable?
For this specific purpose I DO NOT want an array.
You can use a simple one-liner, quite straightforward, though more expensive:
master=$(set | grep -E '^(A|AA|A[A-D][A-D])=' | sort | cut -f2- -d= | tr -d '\n')
set lists all the variables in var=name format
grep filters out the variables we need
sort puts them in the right order (probably optional since set gives a sorted output)
cut extracts the values, removing the variable names
tr removes the newlines
Let's test it.
A=1
AA=2
AAA=3
AAB=4
AAC=5
AAD=6
AAAA=99 # just to make sure we don't pick this one up
master=$(set | grep -E '^(A|AA|A[A-D][A-D])=' | sort | cut -f2- -d= | tr -d '\n')
echo "$master"
Output:
123456
With my best guess, how about:
#!/bin/bash
A=('blah')
AA=('blah2')
AAA=('blah3')
AAB=('blah4')
AAC=('blah5')
# to be continued ..
for varname in A AA A{A..D}{A..Z}; do
value=${!varname}
if [ -n "$value" ]; then
MASTER+=$value
fi
done
echo $MASTER
which yields:
blahblah2blah3blah4blah5...
Although I'm not sure whether this is what the OP wants.
echo {a..z}{a..z}{a..z} | tr ' ' '\n' | head -n 100 | tail -n 3
adt
adu
adv
tells us, that it would go from AAA to ADV to reach 100, or for ADY for 103.
echo A{A..D}{A..Z} | sed 's/ /}${/g'
AAA}${AAB}${AAC}${AAD}${AAE}${AAF}${AAG}${AAH}${AAI}${AAJ}${AAK}${AAL}${AAM}${AAN}${AAO}${AAP}${AAQ}${AAR}${AAS}${AAT}${AAU}${AAV}${AAW}${AAX}${AAY}${AAZ}${ABA}${ABB}${ABC}${ABD}${ABE}${ABF}${ABG}${ABH}${ABI}${ABJ}${ABK}${ABL}${ABM}${ABN}${ABO}${ABP}${ABQ}${ABR}${ABS}${ABT}${ABU}${ABV}${ABW}${ABX}${ABY}${ABZ}${ACA}${ACB}${ACC}${ACD}${ACE}${ACF}${ACG}${ACH}${ACI}${ACJ}${ACK}${ACL}${ACM}${ACN}${ACO}${ACP}${ACQ}${ACR}${ACS}${ACT}${ACU}${ACV}${ACW}${ACX}${ACY}${ACZ}${ADA}${ADB}${ADC}${ADD}${ADE}${ADF}${ADG}${ADH}${ADI}${ADJ}${ADK}${ADL}${ADM}${ADN}${ADO}${ADP}${ADQ}${ADR}${ADS}${ADT}${ADU}${ADV}${ADW}${ADX}${ADY}${ADZ
The final cosmetics is easily made by hand.
One-liner using a for loop:
for n in A AA A{A..D}{A..Z}; do str+="${!n}"; done; echo ${str}
Output:
blahblah2blah3blah4blah5
Say you have the input file inputfile.txt with arbitrary variable names and values:
name="Joe"
last="Doe"
A="blah"
AA="blah2
then do:
master=$(eval echo $(grep -o "^[^=]\+" inputfile.txt | sed 's/^/\$/;:a;N;$!ba;s/\n/$/g'))
This will concatenate the values of all variables in inputfile.txt into master variable. So you will have:
>echo $master
JoeDoeblahblah2
I want to read lines from a text file and save them in a variable.
cat ${1} | while read name; do
namelist=${name_list},${name}
done
the file looks like this:
David
Kevin
Steve
etc.
and i want to get this output instead
David, Kevin, Steve etc.
and save it to the variable ${name_list}
The command:
$ tr -s '\n ' ',' < sourcefile.txt # Replace newlines and spaces with [,]
This will likely return a , as the last character (and potentially the first).
To shave of the comma(s) and return a satisfying result:
$ name_list=$(tr -s '\n ' ',' < sourcefile.txt) # store the previous result
$ name_list=${tmp%,} # shave off the last comma
$ name_list=${tmp#,} # shave off any first comma
EDIT
This solution runs 44% faster and yields consistent and valid results across all Unix platforms.
# This solution
python -mtimeit -s 'import subprocess' "subprocess.call('tmp=$(tr -s "\n " "," < input.txt);echo ${tmp%,} >/dev/null',shell = True)"
100 loops, best of 3: 3.71 msec per loop
# Highest voted:
python -mtimeit -s 'import subprocess' "subprocess.call('column input.txt | sed "s/\t/,/g" >/dev/null',shell = True)"
100 loops, best of 3: 6.69 msec per loop
name_list=""
for name in `cat file.txt`
do VAR="$name_list,$i"
done
EDIT: this script leaves a "," at the beginning of name_list. There are a number of ways to fix this. For example, in bash this should work:
name_list=""
for name in `cat file.txt`; do
if [[ -z $name_list ]]; then
name_list="$i"
else
name_list="$name_list,$i"
fi
done
RE-EDIT: so, thanks to the legitimate complaints of Fredrik:
name_list=""
while read name
do
if [[ -z $name_list ]]; then
name_list="$name"
else
name_list="$name_list,$name"
fi
done < file.txt
Using column, and sed:
namelist=$(column input | sed 's/\t/,/g')
variable=`perl -lne 'next if(/^\s*$/);if($a){$a.=",$_"}else{$a=$_};END{print $a}' your_file`
I have this file:
100: pattern1
++++++++++++++++++++
1:pattern2
9:pattern2
+++++++++++++++++++
79: pattern1
61: pattern1
+++++++++++++++++++
and I want to sort it like this:
++++++++++++++++++++
1:pattern2
9:pattern2
+++++++++++++++++++
61:pattern1
79:pattern1
100:pattern1
+++++++++++++++++++
Is it possible using Linux sort command only ?
If I had :
4:pat1
3:pat2
2:pat2
1:pat1
O/p should be:
1:pat1
++++++++++++
2:pat2
3:pat2
++++++++++++
4:pat1
So, want to sort on first group, but "group" on the pattern of second group.
Please note, the thing after : is a regex pattern not a literal.
Best you can do is to sort it according to the numerical values. But you cannot do anything with the "+"-string.
$ sort -n input
+++++++++++++++++++
+++++++++++++++++++
++++++++++++++++++++
1:wow
9:wow
61: this is it
79: this is it
100: this is it
I don't believe sort alone can do what you need.
Create a new shell script and put this in its contents (ie mysort.sh):
#!/bin/sh
IFS=$'\n' # This makes the for loop below split on newline instead of whitespace.
delim=+++++++++++++++++++
for l in `grep -v ^+| sort -g` # Ignore all + lines and sort by number
do
current=`echo $l | sed s/^[0-9]*://g` # Get what comes after the number
if [ ! -z "$prev" ] && [ "$prev" != "$current" ] # If it has changed...
then # then output a ++++ delimiter line.
echo $delim
fi
prev=$current
echo $l # Output this line.
done
To use it, pipe in the contents of your file like so:
cat input | sh mysort.sh
Probably not -- it's not in the sort of format sort(1) expects. And if you did it would be one of those amazing hacks, not easily used. If you have some sort of rule for what goes between the lines of plus signs, you can do it readily enough with an AWK or Perl or Python script.
If your input was space delimited, not ':' delimited:
sort -rk2 | uniq -D -f1
will do the grouping;
I guess you'd need to sort the 'subsections' later (unfortunately my sort(1) doesn't do composite key ordering. I do believe there are version that allow you to do sort -k2,1n and you'd be done at once).
use --all-repeated=separate instead of -D to get blank separators between groups. Look at man uniq for more ideas!
However, since your input is colon delimited, a hack is required:
sed 's/\([0123456789]\+\):/\1 /' t | sort -rk2 | uniq -D -f1
HTH
Suppose you have a file that contains IP addresses, one address in each line:
10.0.10.1
10.0.10.1
10.0.10.3
10.0.10.2
10.0.10.1
You need a shell script that counts for each IP address how many times it appears in the file. For the previous input you need the following output:
10.0.10.1 3
10.0.10.2 1
10.0.10.3 1
One way to do this is:
cat ip_addresses |uniq |while read ip
do
echo -n $ip" "
grep -c $ip ip_addresses
done
However it is really far from being efficient.
How would you solve this problem more efficiently using bash?
(One thing to add: I know it can be solved from perl or awk, I'm interested in a better solution in bash, not in those languages.)
ADDITIONAL INFO:
Suppose that the source file is 5GB and the machine running the algorithm has 4GB. So sort is not an efficient solution, neither is reading the file more than once.
I liked the hashtable-like solution - anybody can provide improvements to that solution?
ADDITIONAL INFO #2:
Some people asked why would I bother doing it in bash when it is way easier in e.g. perl. The reason is that on the machine I had to do this perl wasn't available for me. It was a custom built linux machine without most of the tools I'm used to. And I think it was an interesting problem.
So please, don't blame the question, just ignore it if you don't like it. :-)
sort ip_addresses | uniq -c
This will print the count first, but other than that it should be exactly what you want.
The quick and dirty method is as follows:
cat ip_addresses | sort -n | uniq -c
If you need to use the values in bash you can assign the whole command to a bash variable and then loop through the results.
PS
If the sort command is omitted, you will not get the correct results as uniq only looks at successive identical lines.
for summing up multiple fields, based on a group of existing fields, use the example below : ( replace the $1, $2, $3, $4 according to your requirements )
cat file
US|A|1000|2000
US|B|1000|2000
US|C|1000|2000
UK|1|1000|2000
UK|1|1000|2000
UK|1|1000|2000
awk 'BEGIN { FS=OFS=SUBSEP="|"}{arr[$1,$2]+=$3+$4 }END {for (i in arr) print i,arr[i]}' file
US|A|3000
US|B|3000
US|C|3000
UK|1|9000
The canonical solution is the one mentioned by another respondent:
sort | uniq -c
It is shorter and more concise than what can be written in Perl or awk.
You write that you don't want to use sort, because the data's size is larger than the machine's main memory size. Don't underestimate the implementation quality of the Unix sort command. Sort was used to handle very large volumes of data (think the original AT&T's billing data) on machines with 128k (that's 131,072 bytes) of memory (PDP-11). When sort encounters more data than a preset limit (often tuned close to the size of the machine's main memory) it sorts the data it has read in main memory and writes it into a temporary file. It then repeats the action with the next chunks of data. Finally, it performs a merge sort on those intermediate files. This allows sort to work on data many times larger than the machine's main memory.
cat ip_addresses | sort | uniq -c | sort -nr | awk '{print $2 " " $1}'
this command would give you desired output
Solution ( group by like mysql)
grep -ioh "facebook\|xing\|linkedin\|googleplus" access-log.txt | sort | uniq -c | sort -n
Result
3249 googleplus
4211 linkedin
5212 xing
7928 facebook
It seems that you have to either use a big amount of code to simulate hashes in bash to get linear behavior or stick to the quadratic superlinear versions.
Among those versions, saua's solution is the best (and simplest):
sort -n ip_addresses.txt | uniq -c
I found http://unix.derkeiler.com/Newsgroups/comp.unix.shell/2005-11/0118.html. But it's ugly as hell...
I feel awk associative array is also handy in this case
$ awk '{count[$1]++}END{for(j in count) print j,count[j]}' ips.txt
A group by post here
You probably can use the file system itself as a hash table. Pseudo-code as follows:
for every entry in the ip address file; do
let addr denote the ip address;
if file "addr" does not exist; then
create file "addr";
write a number "0" in the file;
else
read the number from "addr";
increase the number by 1 and write it back;
fi
done
In the end, all you need to do is to traverse all the files and print the file names and numbers in them. Alternatively, instead of keeping a count, you could append a space or a newline each time to the file, and in the end just look at the file size in bytes.
Most of the other solutions count duplicates. If you really need to group key value pairs, try this:
Here is my example data:
find . | xargs md5sum
fe4ab8e15432161f452e345ff30c68b0 a.txt
30c68b02161e15435ff52e34f4fe4ab8 b.txt
30c68b02161e15435ff52e34f4fe4ab8 c.txt
fe4ab8e15432161f452e345ff30c68b0 d.txt
fe4ab8e15432161f452e345ff30c68b0 e.txt
This will print the key value pairs grouped by the md5 checksum.
cat table.txt | awk '{print $1}' | sort | uniq | xargs -i grep {} table.txt
30c68b02161e15435ff52e34f4fe4ab8 b.txt
30c68b02161e15435ff52e34f4fe4ab8 c.txt
fe4ab8e15432161f452e345ff30c68b0 a.txt
fe4ab8e15432161f452e345ff30c68b0 d.txt
fe4ab8e15432161f452e345ff30c68b0 e.txt
GROUP BY under bash
Regarding this SO thread, there are some different answer regarding different needs.
1. Counting IP as SO request (GROUP BY IP address).
As IP are easy to convert to single integer, for small bunch of address, if you need to repeat this kind of operation many time, using a pure bash function could be a lot more efficient!
Pure bash (no fork!)
There is a way, using a bash function. This way is very quick as there is no fork!...
countIp () {
local -a _ips=(); local _a
while IFS=. read -a _a ;do
((_ips[_a<<24|${_a[1]}<<16|${_a[2]}<<8|${_a[3]}]++))
done
for _a in ${!_ips[#]} ;do
printf "%.16s %4d\n" \
$(($_a>>24)).$(($_a>>16&255)).$(($_a>>8&255)).$(($_a&255)) ${_ips[_a]}
done
}
Note: IP addresses are converted to 32bits unsigned integer value, used as index for array. This use simple bash arrays!
time countIp < ip_addresses
10.0.10.1 3
10.0.10.2 1
10.0.10.3 1
real 0m0.001s
user 0m0.004s
sys 0m0.000s
time sort ip_addresses | uniq -c
3 10.0.10.1
1 10.0.10.2
1 10.0.10.3
real 0m0.010s
user 0m0.000s
sys 0m0.000s
On my host, doing so is a lot quicker than using forks, upto approx 1'000 addresses, but take approx 1 entire second when I'll try to sort'n count 10'000 addresses.
2. GROUP BY duplicates (files content)
By using checksum you could indentfy duplicate files somewhere:
find . -type f -exec sha1sum {} + |
sort |
sed '
:a;
$s/^[^ ]\+ \+//;
N;
s/^\([^ ]\+\) \+\([^ ].*\)\n\1 \+\([^ ].*\)$/\1 \2\o11\3/;
ta;
s/^[^ ]\+ \+//;
P;
D;
ba
'
This will print all duplicates, by line, separated by Tabulation ($'\t' or octal 011 ou could change /\1 \2\o11\3/; by /\1 \2|\3/; for using | as separator).
./b.txt ./e.txt
./a.txt ./c.txt ./d.txt
Could be written as (with | as separator):
find . -type f -exec sha1sum {} + | sort | sed ':a;$s/^[^ ]\+ \+//;N;
s/^\([^ ]\+\) \+\([^ ].*\)\n\1 \+\([^ ].*\)$/\1 \2|\3/;ta;s/^[^ ]\+ \+//;P;D;ba'
Pure bash way
By using nameref, you could build bash arrays holding all duplicates:
declare -iA sums='()'
while IFS=' ' read -r sum file ;do
declare -n list=_LST_$sum
list+=("$file")
sums[$sum]+=1
done < <(
find . -type f -exec sha1sum {} +
)
From there, you have a bunch of arrays holding all duplicates file name as separated element:
for i in ${!sums[#]};do
declare -n list=_LST_$i
printf "%d %d %s\n" ${sums[$i]} ${#list[#]} "${list[*]}"
done
This may output something like:
2 2 ./e.txt ./b.txt
3 3 ./c.txt ./a.txt ./d.txt
Where count of files by md5sum (${sums[$shasum]}) match count of element in arrays ${_LST_ShAsUm[#]}.
for i in ${!sums[#]};do
declare -n list=_LST_$i
echo ${list[#]#A}
done
declare -a _LST_22596363b3de40b06f981fb85d82312e8c0ed511=([0]="./e.txt" [1]="./b.txt")
declare -a _LST_f572d396fae9206628714fb2ce00f72e94f2258f=([0]="./c.txt" [1]="./a.txt" [2]="./d.txt")
Note that this method could handle spaces and special characters in filenames!
3. GROUP BY columns in a table
As efficient sample using awk was provided by Anonymous, here is a pure bash solution.
So you want to sumarize columns 3 to last column and group by columns 1 and 2, having table.txt looking like
US|A|1000|2000
US|B|1000|2000
US|C|1000|2000
UK|1|1000|2000
UK|1|1000|2000|3000
UK|1|1000|2000|3000|4000
For not too big tables, you could:
myfunc() {
local -iA restabl='()';
local IFS=+
while IFS=\| read -ra ar; do
restabl["${ar[0]}|${ar[1]}"]+="${ar[*]:2}"
done
for i in ${!restabl[#]} ;do
printf '%s|%s\n' "$i" "${restabl[$i]}"
done
}
Could ouput something like:
myfunc <table.txt
UK|1|19000
US|A|3000
US|C|3000
US|B|3000
And to have table sorted:
myfunc() {
local -iA restabl='()';
local IFS=+ sorted=()
while IFS=\| read -ra ar; do
sorted[64#${ar[0]}${ar[1]}]="${ar[0]}|${ar[1]}"
restabl["${ar[0]}|${ar[1]}"]+="${ar[*]:2}"
done
for i in ${sorted[#]} ;do
printf '%s|%s\n' "$i" "${restabl[$i]}"
done
}
Must return:
myfunc <table
UK|1|19000
US|A|3000
US|B|3000
US|C|3000
I'd have done it like this:
perl -e 'while (<>) {chop; $h{$_}++;} for $k (keys %h) {print "$k $h{$k}\n";}' ip_addresses
but uniq might work for you.
Importing data to sqlite db and using sql syntax (just an other idea).
I know it's too much for this example but would be useful for complex queries with multiple files (tables)
#!/bin/bash
trap clear_db EXIT
clear_db(){ rm -f "mydb$$"; }
# add header to input_file (IP)
INPUT_FILE=ips.txt
# import file into db
sqlite3 -csv mydb$$ ".import ${INPUT_FILE} mytable"
# using sql statements on table 'mytable'
sqlite3 mydb$$ -separator " " "SELECT IP, COUNT(*) FROM mytable GROUP BY IP;"
10.0.10.1 3
10.0.10.2 1
10.0.10.3 1
I understand you are looking for something in Bash, but in case someone else might be looking for something in Python, you might want to consider this:
mySet = set()
for line in open("ip_address_file.txt"):
line = line.rstrip()
mySet.add(line)
As values in the set are unique by default and Python is pretty good at this stuff, you might win something here. I haven't tested the code, so it might be bugged, but this might get you there. And if you want to count occurrences, using a dict instead of a set is easy to implement.
Edit:
I'm a lousy reader, so I answered wrong. Here's a snippet with a dict that would count occurences.
mydict = {}
for line in open("ip_address_file.txt"):
line = line.rstrip()
if line in mydict:
mydict[line] += 1
else:
mydict[line] = 1
The dictionary mydict now holds a list of unique IP's as keys and the amount of times they occurred as their values.
This does not answer the count element of the original question, but this question is the first search engine result when searching for what I wanted to achieve, so I thought this may help someone as it relates to 'group by' functionality.
I wanted to order files based on groupings of them, where the presence of some string in the filename determined the group.
It uses a temporary grouping/ordering prefix which is removed after ordering; sed substitute expressions (s#pattern#replacement#g) match the target string and prepend an integer to the line corresponding to the desired sort order of that target string. Then, grouping prefix is removed with cut.
Note that the sed expressions could be joined (e.g. sed -e '<expr>; <expr>; <expr>;') but here they're split for readability.
It's not pretty and probably not fast (I'm dealing with <50 items) but it at-least conceptually simple and doesn't require learning awk.
#!/usr/bin/env bash
for line in $(find /etc \
| sed -E -e "s#^(.*${target_string_A}.*)#${target_string_A_sort_index}:\1#;" \
| sed -E -e "s#^(.*${target_string_B}.*)#${target_string_B_sort_index}:\1#;" \
| sed -E -e "s#^/(.*)#00:/\1#;" \
| sort \
| cut -c4-
)
do
echo "${line}"
done
e.g. Input
/this/is/a/test/a
/this/is/a/test/b
/this/is/a/test/c
/this/is/a/special/test/d
/this/is/a/another/test/e
#!/usr/bin/env bash
for line in $(find /etc \
| sed -E -e "s#^(.*special.*)#10:\1#;" \
| sed -E -e "s#^(.*another.*)#05:\1#;" \
| sed -E -e "s#^/(.*)#00:/\1#;" \
| sort \
| cut -c4-
)
do
echo "${line}"
done
/this/is/a/test/a
/this/is/a/test/b
/this/is/a/test/c
/this/is/a/another/test/e
/this/is/a/special/test/d
A combination of awk + sort (with version sort flag) is probably fastest (if ur environment has awk at all):
echo "${input...}" |
{m,g}awk '{ __[$+_]++ } END { for(_ in __) { print "",+__[_],_ } }' FS='^$' OFS='\t' |
gsort -t$'\t' -k 3,3 -V
Only the post GROUP-BY summary rows are being sent to the sorting utility - which is far less system intensive sort compared to pre-sorting the input rows for no reason.
For small inputs, e.g. fewer than 1000 rows or so, just directly sort|uniq -c it.
3 10.0.10.1
1 10.0.10.2
1 10.0.10.3
Sort may be omitted if order is not significant
uniq -c <source_file>
or
echo "$list" | uniq -c
if the source list is a variable