I have an input file that looks somewhat like this:
PlayerId,Name,Score1,Score2
1,A,40,20
2,B,30,10
3,C,25,28
I want to write an awk command that checks for players with sum of scores greater than 50 and outputs the PlayerId,and PlayerName in sorted order of their total score.
When I try the following:
awk 'BEGIN{FS=",";}{$5=$3+$4;if($5>50) print $1,$2}' | sort -k5
It does not work and seemingly sorts them on the basis of their ids.
1 A
3 C
Whereas the correct output I'm expecting is : ( since Player A has sum of scores=60, and C has sum of scores=53, and we want the output to be sorted in ascending order )
3 C
1 A
In addition to this,what confuses me a bit is when I try to sort it on the basis of score1, i.e. column 3 but intend to print only the corresponding ids and names, it dosen't work either.
awk 'BEGIN{FS=",";}{$5=$3+$4;if($5>50) print $1,$2}' | sort -k3
And outputs :
1 A
3 C
But if the $3 with respect to what the data is being sorted is included in the print,
awk 'BEGIN{FS=",";}{$5=$3+$4;if($5>50)print $1,$2,$3}' | sort -k3
It produces the correct output ( but includes the unwanted score1 parameter in display )
3 C 25
1 A 40
But what if one wants to only print the id and name fields ?
Actually I'm new to awk commands, and probably I'm not using the sort command correctly. It would be really helpful if someone could explain.
I think this is what you're trying to do:
$ awk 'BEGIN{FS=","} {sum=$3+$4} sum>50{print sum,$1,$2}' file |
sort -k1,1n | cut -d' ' -f2-
3 C
1 A
You have to print the sum so you can sort by it and then the cut removes it.
If you wanted the header output too then it'd be:
$ awk 'BEGIN{FS=","} {sum=$3+$4} (NR==1) || (sum>50){print (NR>1),sum,$1,$2}' file |
sort -k1,2n | cut -d' ' -f3-
PlayerId Name
3 C
1 A
if you outsource sorting, you need to have the auxiliary values and need to cut it out later, some complication is due to preserve the header.
$ awk -F, 'NR==1 {print s "\t" $1 FS $2; next}
(s=$3+$4)>50 {print s "\t" $1 FS $2 | "sort -n" }' file | cut -f2
PlayerId,Name
3,C
1,A
Related
Student details are stored in a file system as follows:
Roll_no,name,socre1,score2
101,ABC,50,55
102,XYZ,48,54
103,CWE,42,34
104,ZSE,65,72
105,FGR,31,45
106,QWE,68,45
Q.Write the unix command to display Roll_no and name of the student whose total score is greater than 100 the student details are to be displayed sorted in descending order of the total score.
total score as to be calculated as follows :-
totalscore=score1+score2
file also content the header(Roll_no,name,socre1,score2)
My solution:
awk 'BEGIN {FS=",";OFS=" "} {if(NR>1){if($3+$4>100){s[$1]=$2}}} END{for (i in s) {print i,h[i]}}' stu.txt| sort -rk 2n
I am not getting how to get sorting according to total score?
please help guys!
output:-
104 ZSE
106 QWE
101 ABC
102 XYZ
Could you please try following. To keep it simple in calculation(1st get total of numbers for all lines which are greater than 100 Then sort it reverse order by total as per OP then print only first 2 columns by cut)
awk 'BEGIN{FS=OFS=","} $3+$4>100{print $1,$2,$3+$4}' Input_file |
sort -t, -nr -k3 |
cut -d',' -f 1-2
OR in case you want output in space delimiter in output then try following.
awk 'BEGIN{FS=","} $3+$4>100{print $1,$2,$3+$4}' Input_file |
sort -nr -k3 |
cut -d' ' -f 1-2
Explanation: Adding detailed explanation for above.
awk 'BEGIN{FS=OFS=","} $3+$4>100{print $1,$2,$3+$4}' Input_file | ##Starting awk program setting FS, OFS as comma. Then checking 3rd+4th column sum is greater than 100 then printing 1st, 2nd field along with sum of 3rd and 4th field here. Now passing its output as input to next command.
sort -t, -nr -k3 | ##Sorting output with setting delimiter as comma and sorting it reverse order witg 3rd column here, sending output as input to next command.
cut -d',' -f 1-2 ##Getting first 2 fields by setting delimiter comma here, to get name and roll number here.
OR
sort -t, -nr -k3 < <(awk 'BEGIN{FS=OFS=","} $3+$4>100{print $1,$2,$3+$4}' Input_file) |
cut -d',' -f 1-2
OR in case you need output as space delimited then try following.
sort -nr -k3 < <(awk 'BEGIN{FS=","} $3+$4>100{print $1,$2,$3+$4}' Input_file) |
cut -d' ' -f 1-2
$ awk 'BEGIN {OFS=FS=","}
NR==1 {print $0, "total"; next}
{if(($5=$3+$4)>100) print | "sort -t, -k5nr"}' file
Roll_no,name,socre1,score2,total
104,ZSE,65,72,137
106,QWE,68,45,113
101,ABC,50,55,105
102,XYZ,48,54,102
without header and individual scores
$ awk 'BEGIN{OFS=FS=","}
NR>1 && ($3+=$4)>100{print $1,$2,$3}' file | sort -t, -k3nr
104,ZSE,137
106,QWE,113
101,ABC,105
102,XYZ,102
or
$ awk 'BEGIN{OFS=FS=","}
NR>1 && ($3+=$4)>100 && NF--' file | sort -t, -k3nr
104,ZSE,137
106,QWE,113
101,ABC,105
102,XYZ,102
without the final score and not comma delimited
$ awk -F, 'NR>1 && ($3+=$4)>100 && NF--' file | sort -k3nr | cut -d' ' -f1,2
104 ZSE
106 QWE
101 ABC
102 XYZ
reads as written
if line number is greater than one (skip header) AND
if field 3 + field 4 > 100 (assigned back to field 3) then
if both conditions are satisfied decrement field count so that last field won't be printed.
sort the results based on the third field,
remove the last field.
you were close:
awk 'BEGIN {FS=OFS=","} {if(NR>1){if($3+$4>100){s[$1]=$2}}} END{for (i in s) {print i,s[i]}}' stu.txt| sort -rk 2n
I'm learning AWK to read a big file which format is similar to this MasterFile:
Beth|4.00|0|
Dan|3.75|0|
Kathy|4.00|10|
Mark|5.00|20|
Mary|5.50|22|
Susie|4.25|18|
Jise|5.62|0|
Mark|5.60|23.3|
Mary|8.50|42|
Susie|8.75|8.8|
Jise|3.62|0.8|
Beth|3.21|10|
Dan|8.39|20|
I would like to sample by unique values (size K) from the first column with size N (I choose it).
What I have done is following: I select unique values from first column and save it as IDfile.txt. Later, I take K random values from that archive and I match it with the MasterFile. I mean:
awk -F\| 'BEGIN{srand()}{print rand() " " $0}' IDfile | sort -n | tail -n K| awk -F'[[:blank:]|]+' 'BEGIN{OFS="|"}{$1="";sub(/\|/,"")}'1>tmp | awk -F\| 'NR==FNR{a[$1];next} {for (i in a) if(index($0,i)) print $0}' tmp MasterFile
But the output has repeated values and the result that I'd like to get is like to (assuming that K=3):
Beth|4.00|0|
Mark|5.60|23.3|
Mary|5.50|22|
I know that my code is far from efficient [or nice] and I'm open to suggestions [].
Thanks!
this is the one of the right ways to do this
$ sort -t'|' -u -k1,1 file | shuf -n3
Mark|5.00|20|
Kathy|4.00|10|
Jise|5.62|0|
change -n3 to whatever number of unique entries you need.
This statement gives me the count of unique values in column 1:
awk -F ',' '{print $1}' infile1.csv | sort | uniq -c | sort -nr > outfile1.csv
It does what I expected (gives the count (left) of unique values (right) in the column):
117 5
58 0
18 4
14 3
11 1
9 2
However, now I want to create a loop, so it will go through all columns.
I tried:
for i in {1..10}
do
awk -F ',' '{print $$i}' infile.csv | sort | uniq -c | sort -nr > outfile$i.csv
done
This does not do the job (it does produce a file but with much more data). I think that a variable in a print statement, as I tried with print $$i, is not something that works in general, since I did not come across it so far.
I also tried this:
awk -F ',' '{for(i=1;i<=NF;i++) infile.csv | sort | uniq -c | sort -nr}' > outfile$i.csv
But this does not give any result at all (meaning syntax errors for infile and sort command). I am sure I am using the for statement the wrong way.
Ideally, I would like the code to find the count of unique values for each column and print them all in the same output file. However, I am already very happy with a well functioning loop.
Please let me know if this explanation is not good enough, I will do my best to clarify.
Any time you write a loop in shell just to manipulate text you have the wrong approach. Just do it in one awk command, something like this using GNU awk for 2D arrays and sorted in (untested since you didn't provide any sample input):
awk -F, '
BEGIN { PROCINFO["sorted_in"] = "#val_num_desc" }
{ for (i=1; i<=NF; i++) cnt[i][$i]++ }
END {
for (i=1; i<=NF; i++)
for (val in cnt[i])
print val, cnt[i][val] > ("outfile" i ".csv")
}
' infile.csv
No need for half a dozen different commands, pipes, etc.
You want to loop through the columns and perform the same command in each one of them. So what you are doing is fine: pass the column name to awk. However, you need to pass the value differently, so that it is an awk variable:
for i in {1..10}
do
awk -F ',' -v col=$i '{print $col}' infile.csv | sort | uniq -c | sort -nr > outfile$i.csv
^^^^^^^^^^^^^^^^^^^^^^^^
done
Say there is an input file with tabs delimited field, the first field is integer
1 abc
1 def
1 ghi
1 lalala
1 heyhey
2 ahb
2 bbh
3 chch
3 chchch
3 oiohho
3 nonon
3 halal
3 whatever
First, i need to compute the counts of the unique values in the first field, that will be:
5 for 1, 2 for 2, and 6 for 3
Then I need to find the max of these counts, in this case, it's 6.
Now i need to pass "6" to another awk script as a parmeter.
I know i can use command below to get a list of count:
cut -f1 input.txt | sort | uniq -c | awk -F ' ' '{print $1}' | sort
but how do i get the first count number and pass it to the next awk command as a parameter not as an input file?
This is nothing very specific for awk.
Either a program can read from stdin, then you can pass the input with a pipe:
prg1 | prg2
or your program expects input as parameter, then you use
prg2 $(prg1)
Note that in both cases prg1 is processed before prg2.
Some programs allow both possibilities, while a huge amount of data is rarely passed as argument.
This AWK script replaces your whole pipeline:
awk -v parameter="$(awk '{a[$1]++} END {for (i in a) {if (a[i] > max) {max = a[i]}}; print max}' inputfile)" '{print parameter}' otherfile
where '{print parameter}' is a standin for your other AWK script and "otherfile" is the input for that script.
Note: It is extremely likely that the two AWK scripts could be combined into one which would be less of a hack than doing it in a way such as that outlined in your question (awk feeding awk).
You can use the shell's $() command substitution:
awk -f script -v num=$(cut -f1 input.txt | sort | uniq -c | awk -F ' ' '{print $1}' | sort | tail -1) < input_file
(I added the tail -1 to ensure that at most one line is used.)
The given file is in the below format.
GGRPW,33332211,kr,P,SUCCESS,systemrenewal,REN,RAMS,SAA,0080527763,on:X,10.0,N,20120419,migr
GBRPW,1232221,uw,P,SUCCESS,systemrenewal,REN,RAMS,ASD,20075578623,on:X,1.0,N,20120419,migr
GLSH,21122111,uw,P,SUCCESS,systemrenewal,REN,RAMS,ASA,0264993503,on:X,10.0,N,20120419,migr
I need to take out duplicates and count(each duplicates categorized by f1,2,5,14). Then insert into database with the first duplicate occurence record entire fields and tag the count(dups) in another column. For this I need to cut all the 4 mentioned fields and sort and find the dups using uniq -d and for counts I used -c. Now again coming back after all sorting out of dups and it counts I need the output to be in the below form.
3,GLSH,21122111,uw,P,SUCCESS,systemrenewal,REN,RAMS,ASA,0264993503,on:X,10.0,N,20120419,migr
Whereas three being the number of repeated dups for f1,2,5,14 and rest of the fields can be from any of the dup rows.
By this way dups should be removed from the original file and show in the above format.
And the remaining in the original file will be uniq ones they go as it is...
What I have done is..
awk '{printf("%5d,%s\n", NR,$0)}' renewstatus_2012-04-19.txt > n_renewstatus_2012-04-19.txt
cut -d',' -f2,3,6,15 n_renewstatus_2012-04-19.txt |sort | uniq -d -c
but this needs a point back again to the original file to get the lines for the dup occurences. ..
let me not confuse.. this needs a different point of view.. and my brain is clinging on my approach.. need a cigar..
Any thots...??
sort has an option -k
-k, --key=POS1[,POS2]
start a key at POS1, end it at POS2 (origin 1)
uniq has an option -f
-f, --skip-fields=N
avoid comparing the first N fields
so sort and uniq with field numbers(count NUM and test this cmd yourself, plz)
awk -F"," '{print $0,$1,$2,...}' file.txt | sort -k NUM,NUM2 | uniq -f NUM3 -c
Using awk's associative arrays is a handy way to find unique/duplicate rows:
awk '
BEGIN {FS = OFS = ","}
{
key = $1 FS $2 FS $5 FS $14
if (key in count)
count[key]++
else {
count[key] = 1
line[key] = $0
}
}
END {for (key in count) print count[key], line[key]}
' filename
SYNTAX :
awk -F, '!(($1 SUBSEP $2 SUBSEP $5 SUBSEP $14) in uniq){uniq[$1,$2,$5,$14]=$0}{count[$1,$2,$5,$14]++}END{for(i in count){if(count[i] > 1)file="dupes";else file="uniq";print uniq[i],","count[i] > file}}' renewstatus_2012-04-19.txt
Calculation:
sym#localhost:~$ cut -f16 -d',' uniq | sort | uniq -d -c
124275 1 -----> SUM OF UNIQ ( 1 )ENTRIES
sym#localhost:~$ cut -f16 -d',' dupes | sort | uniq -d -c
3860 2
850 3
71 4
7 5
3 6
sym#localhost:~$ cut -f16 -d',' dupes | sort | uniq -u -c
1 7
10614 ------> SUM OF DUPLICATE ENTRIES MULTIPLIED WITH ITS COUNTS
sym#localhost:~$ wc -l renewstatus_2012-04-19.txt
134889 renewstatus_2012-04-19.txt ---> TOTAL LINE COUNTS OF THE ORIGINAL FILE, MATCHED EXACTLY WITH (124275+10614) = 134889