I use sort -r | head and get the out put like this:
8 a1
8 a2
5 a3
5 a4
4 a5
4 a6
4 a7
4 a8
4 a9
4 a0
What can I do to make the output like this:
8 a1
8 a2
only the largest k1 number show up????
There are several ways to do it, but here is one using awk. Since it is already sorted, you want to check to just print lines that match the first value by piping the headed list into something like
awk 'BEGIN{maxval=0}; (maxval==0) {maxval=$1}; ($1==maxval) {print $0}'
Related
Imagine we have a txt file like the next one:
Input:
a1 D1
b1 D1
c1 D1
a1 D2
a1 D3
c1 D3
I want to count the time each element in the first column appears but also keep the information provided by the second column (someway). Potential possible output formats are represented, but any coherent alternative is also accepted:
Possible output 1:
3 a1 D1,D2,D3
1 b1 D1
2 c1 D1,D3
Possible output 2:
3 a1 D1
1 b1 D1
2 c1 D1
3 a1 D2
3 a1 D3
1 c1 D3
How can I do this? I guess a combination sort -k 1 input | uniq -c <keep col2> or perhaps using awk but I was not able to write anything that works. However, all answers are considered.
I would harness GNU AWK for this task following way, let file.txt content be
a1 D1
b1 D1
c1 D1
a1 D2
a1 D3
c1 D3
then
awk 'FNR==NR{arr[$1]+=1;next}{print arr[$1],$0}' file.txt file.txt
gives output
3 a1 D1
1 b1 D1
2 c1 D1
3 a1 D2
3 a1 D3
2 c1 D3
Explanation: 2-pass solution (observe that file.txt is repeated), first pass does count number of occurences of first column value storing that data into array arr, second pass is for printing computed number from array, followed by whole line.
(tested in GNU Awk 5.0.1)
Using any awk:
$ awk '
{
vals[$1] = ($1 in vals ? vals[$1] "," : "") $2
cnts[$1]++
}
END {
for (key in vals) {
print cnts[key], key, vals[key]
}
}
' file
3 a1 D1,D2,D3
1 b1 D1
2 c1 D1,D3
Problem
I am writing a bash script and I have an array, where each value consists of two columns. It looks like this:
for i in "${res[#]}"; do
echo "$i"
done
#Stream1
0 a1
1 b1
2 c1
4 d1
6 e1
#Stream2
0 a2
1 b2
3 c2
4 d2
9 f2
...
I would like to combine the output from this array into a larger table, and multiplex the indices. Furthermore, I would like to format the top row by inserting comment #Sec.
I would like the result to be something like this:
#Sec Stream1 Stream2
0 a1 a2
1 b1 b2
2 c1
3 c2
4 d1 d2
6 e1
9 f2
The insertion of #Sec and removal of the # behind the Streamkeyword is not necessary but desired if not too difficult.
Tried Solutions
I have tried piping to column and awk, but have not been able to produce the desired results.
EDIT
resis an array in a bash script. It is quite large, so I will only provide a short selection. Running echo "$( typeset -p res)"produces following output:
declare -a res='([1]="#Stream1
0 3072
1 6144
2 5120
3 1024
5 6144
..." [2]="#Stream2
0 3072
1 5120
2 4096
3 3072
53 3072
55 1024
57 2048")'
As for the 'result', my initial intention was to assign the resulting table to a variable and use it in another awk script to calculate the moving averages for specified indices, and plot the results. This will be done for ~20 different files. However I am open to other solutions.
The number of streams may vary from 10 to 50. Each stream having from 100 to 300 rows.
You may use this awk solution:
cat tabulate.awk
NF == 1 {
h = h OFS substr($1, 2)
++numSec
next
}
{
keys[$1]
map[$1,numSec] = $2
}
END {
print h
for (k in keys) {
printf "%s", k
for (i=1; i<=numSec; ++i)
printf "\t%s", map[k,i]
print ""
}
}
Then use it as:
awk -v OFS='\t' -v h='#Sec' -f tabulate.awk file
#Sec Stream1 Stream2
0 a1 a2
1 b1 b2
2 c1
3 c2
4 d1 d2
6 e1
9 f2
My bash script is calling a awk script that nicely merges two files
mapfieldfile1=1
mapfieldfile2=2
awk -v FS="\t" 'BEGIN {OFS="\t"}
FNR==NR{hash1['"\$${mapfieldfile2}"']=$1 FS $3 FS $4 FS $5 FS $6;next}
('"\$${mapfieldfile1}"' in hash1){ print $0, hash1['"\$${mapfieldfile1}"']}' file2 file1
However I want to a more general version,where I don't have to hardcode the columns that I want to print, I simply want to print everything but my id column. Replacing $1 FS $3 FS $4 FS $5 FS $6 for $0 "almost" does the work, except that repeats the id column. I have been trying to dynamically create a a string similar to the $1 FS $3 FS $4 FS $5 FS $6 but I am getting literally the $1 $3 $4 $5 $6 strings in the merged file, as opposed to expanding their values. Also, smaller side effects: I am adding a tab in the middle and losing some headers, below is the code and example files.
I would like to find the solution to my merge and also understand what I am doing wrong and why my variables are not expanding.
I appreciate any help!
mapfieldfile1=1
mapfieldfile2=2
awk -v FS="\t" 'BEGIN {OFS="\t";strfields=""}
FNR==NR{for(i=1;i<=NF;i++) if(i!='"${mapfieldfile2}"') {strfields=strfields" "FS" $"i};
hash1['"\$${mapfieldfile2}"']=strfields;strfields="";next}
('"\$${mapfieldfile1}"' in hash1){print $0, hash1['"\$${mapfieldfile1}"']}' file2 file1
$cat file1
sampleid s1 s2 s3 s4
1 1 1 1 1
2 2 2 2 2
3 3 3 3 3
4 4 4 4 4
$cat file2
a0 sampleid a1 a2 a3 a4
a0 1 a a a a4
a0 2 b b b a4
a0 3 c c c a4
a0 5 e e e a4
$cat first_code_result.txt (good one!)
sampleid s1 s2 s3 s4 a0 a1 a2 a3 a4
1 1 1 1 1 a0 a a a a4
2 2 2 2 2 a0 b b b a4
3 3 3 3 3 a0 c c c a4
$cat second_code_result.txt
sampleid s1 s2 s3 s4 $1 $3 $4 $5 $6
1 1 1 1 1 $1 $3 $4 $5 $6
2 2 2 2 2 $1 $3 $4 $5 $6
3 3 3 3 3 $1 $3 $4 $5 $6
Try this (untested):
awk -v mf1="$mapfieldfile1" -v mf2="$mapfieldfile2" '
BEGIN {FS=OFS="\t"}
FNR==NR{sub(/\t[^\t]+/,""); hash1[$mf2]=$0; next}
($mf1 in hash1){ print $0, hash1[$mf1]}
' file2 file1
Don't let shell variables expand within awk scripts, use a regexp to remove fields from the record and idk why the script you haven't shown us is printing literally $3, etc. but you must be including them in a string. You'd have to post that script for help debugging it.
Check where mf1 vs mf2 should appear, I got confused reading your scripts.
EDIT - I had to tweak it as above I was deleting $2 before using it:
$ awk -v mf1="1" -v mf2="2" '
BEGIN {FS=OFS="\t"}
FNR==NR{key=$mf2; sub(/\t[^\t]+/,""); hash1[key]=$0; next}
($mf1 in hash1){ print $0, hash1[$mf1]}
' file2 file1
sampleid s1 s2 s3 s4 a0 a1 a2 a3 a4
1 1 1 1 1 a0 a a a a4
2 2 2 2 2 a0 b b b a4
3 3 3 3 3 a0 c c c a4
Note that the sub() above relies on the key field being $2 and FS being a tab. If you need a more general solution let us know.
Here's a version that'll do what you want for any key field values and will work in any awk, it just requires the FS to be a tab or some other fixed string (i.e. not a regexp):
$ cat tst.awk
BEGIN { FS=OFS="\t" }
NR==FNR {
key = $mf2
val = ""
nf = 0
for (i=1; i<=NF; i++) {
if (i != mf2) {
val = (nf++ ? val FS : "") $i
}
}
hash1[key] = val
next
}
$mf1 in hash1 { print $0, hash1[$mf1] }
$ awk -v mf1="1" -v mf2="2" -f tst.awk file2 file1
sampleid s1 s2 s3 s4 a0 a1 a2 a3 a4
1 1 1 1 1 a0 a a a a4
2 2 2 2 2 a0 b b b a4
3 3 3 3 3 a0 c c c a4
if your files are sorted already, the default output of join is what you want
$ join -t$'\t' -11 -22 file1 file2
sampleid s1 s2 s3 s4 a0 a1 a2 a3 a4
1 1 1 1 1 a0 a a a a4
2 2 2 2 2 a0 b b b a4
3 3 3 3 3 a0 c c c a4
or, after prettying with column
$ join -t$'\t' -11 -22 file1 file2 | column -t
sampleid s1 s2 s3 s4 a0 a1 a2 a3 a4
1 1 1 1 1 a0 a a a a4
2 2 2 2 2 a0 b b b a4
3 3 3 3 3 a0 c c c a4
Input File 1
A1 123 AA
B1 123 BB
C2 44 CC1
D1 12 DD1
E1 11 EE1
Input File 2
A sad21 1
DD1 124f2 2
CC 123tges 3
BB 124sdf 4
AA 1asrf 5
Output File
A1 123 AA 1asrf 5
B1 123 BB 124sdf 4
D1 12 DD1 124f2 2
Making of Output file
We check 3rd column of Input File 1 and 1st Col of Input File 2.
If they match , we print it in Output file.
Note :
The files are not sorted
I tried :
join -t, A B | awk -F "\t" 'BEGIN{OFS="\t"} {if ($3==$4) print $1,$2,$3,$4,$6}'
But this doesnot work as files are unsorted. so the condition ($3==$4) won't work all the time. Please help .
nawk 'FNR==NR{a[$3]=$0;next}{if($1 in a){p=$1;$1="";print a[p],$0}}' file1 file2
tested below:
> cat file1
A1 123 AA
B1 123 BB
C2 44 CC1
D1 12 DD1
E1 11 EE1
> cat file2
A sad21 1
DD1 124f2 2
CC 123tges 3
BB 124sdf 4
AA 1asrf 5
> awk 'FNR==NR{a[$3]=$0;next}{if($1 in a){p=$1;$1="";print a[p],$0}}' file1 file2
D1 12 DD1 124f2 2
B1 123 BB 124sdf 4
A1 123 AA 1asrf 5
>
You can use join, but you need to sort on the key field first and tell join that the key in the first file is column 3 (-1 3):
join -1 3 <(sort -k 3,3 file1) <(sort file2)
Will get you the correct fields, output (with column -t for output formatting):
AA A1 123 1asrf 5
BB B1 123 124sdf 4
DD1 D1 12 124f2 2
To get the same column ordering listed in the question, you need to specify the output format:
join -1 3 -o 1.1,1.2,1.3,2.2,2.3 <(sort -k 3,3 file1) <(sort file2)
i.e. file 1 fields 1 through 3 then file 2 fields 2 and 3. Output (again with column -t):
A1 123 AA 1asrf 5
B1 123 BB 124sdf 4
D1 12 DD1 124f2 2
perl -F'/\t/' -anle 'BEGIN{$f=1}if($f==1){$H{$F[2]}=$_;$f++ if eof}else{$l=$H{$F[0]};print join("\t",$l,#F[1..$#F]) if defined$l}' f1.txt f2.txt
or shorter
perl -F'/\t/' -anle'$f?($l=$H{$F[0]})&&print(join"\t",$l,#F[1..$#F]):($H{$F[2]}=$_);eof&&$f++' f1.txt f2.txt
One way using awk:
awk 'BEGIN { FS=OFS="\t" } FNR==NR { array[$1]=$2 OFS $3; next } { if ($3 in array) print $0, array[$3] }' file2.txt file1.txt
Results:
A1 123 AA 1asrf 5
B1 123 BB 124sdf 4
D1 12 DD1 124f2 2
This might work for you (GNU sed):
sed 's|\(\S*\)\(.*\)|/\\s\1$/s/$/\2/p|' file2 | sed -nf - file1
[Update2] As it often happens, the scope of the task expanded quite a bit as a understood it better. The obsolete parts are crossed out, and you find the updated explanation below. [/Update2]
I have a pair of rather large log files with very similar content, except that some strings are different between the two. A couple of examples:
UnifiedClassLoader3#19518cc | UnifiedClassLoader3#d0357a
JBossRMIClassLoader#13c2d7f | JBossRMIClassLoader#191777e
That is, wherever the first file contains UnifiedClassLoader3#19518cc, the second contains UnifiedClassLoader3#d0357a, and so on. [Update] There are about 40 distinct pairs of such identifiers.[/Update]
UnifiedClassLoader3#19518cc | UnifiedClassLoader3#d0357a
JBossRMIClassLoader#13c2d7f | JBossRMIClassLoader#191777e
Logi18n#177060f | Logi18n#12ef4c6
LogFactory$1#15e3dc4 | LogFactory$1#2942da
That is, wherever the first file contains UnifiedClassLoader3#19518cc, the second contains UnifiedClassLoader3#d0357a, and so on. Note that all these strings are inside long lines of text, and they appear in many rows, intermixed with each other. There are about 4000 distinct pairs of such identifiers, and the size of each file is about 34 MB. So performance became an issue as well.
I want to replace these with identical IDs so that I can spot the really important differences between the two files. I.e. I want to replace all occurrences of both UnifiedClassLoader3#19518cc in file1 and UnifiedClassLoader3#d0357a in file2 with UnifiedClassLoader3#1; all occurrences of both Logi18n#177060f in file1 and Logi18n#12ef4c6 in file2 with Logi18n#2 etc. The counters 1 and 2 are arbitrary choices - the only requirement is that there is a one to one mapping between the old and new strings (i.e. the same string is always replaced by the same value and no different strings are replaced by the same value).
Using the Cygwin shell, so far I managed to list all different identifiers occurring in one of the files with
grep -o -e 'ClassLoader[0-9]*#[0-9a-f][0-9a-f]*' file1.log | sort | uniq
grep -o -e '[A-Z][A-Za-z0-9]*\(\$[0-9][0-9]*\)*#[0-9a-f][0-9a-f]*' file1.log
| sort | uniq
However, now the original order is lost, so I don't know which is the pair of which ID in the other file. With grep -n I can get the line number, so the sort would preserve the order of appearance, but then I can't weed out the duplicate occurrences. Unfortunately grep can not print only the first match of a pattern.
I figured I could save the list of identifiers produced by the above command into a file, then iterate over the patterns in the file with grep -n | head -n 1, concatenate the results and sort them again. The result would be something like
2 ClassLoader3#19518cc
137 ClassLoader#13c2d7f
563 ClassLoader3#1267649
...
Then I could (using sed itself) massage this into a sed command like
sed -e 's/ClassLoader3#19518cc/ClassLoader3#2/g'
-e 's/ClassLoader#13c2d7f/ClassLoader#137/g'
-e 's/ClassLoader3#1267649/ClassLoader3#563/g'
file1.log > file1_processed.log
and similarly for file2.
However, before I start, I would like to verify that my plan is the simplest possible working solution to this.
Is there any flaw in this approach? Is there a simpler way?
I think this does the trick, or at least comes close
#!/bin/sh
for PREFIX in file1 file2
do
cp ${PREFIX}.log /tmp/filter.$$.txt
FILE_MAP=`egrep -o -e 'ClassLoader[0-9a-f]*#[0-9a-f]+' ${PREFIX}.log | uniq | egrep -n .`
for MAP in `echo $FILE_MAP`
do
NUMBER=`echo $MAP | cut -d : -f 1`
WORD=`echo $MAP | cut -d : -f 2`
sed -e s/$WORD/ClassLoader#$NUMBER/g /tmp/filter.$$.txt > ${PREFIX}_processed.log
cp ${PREFIX}_processed.log /tmp/filter.$$.txt
done
rm /tmp/filter.$$.txt
done
Let me know if you have questions on how it works and why.
Here's my test data and the output
file1.log:
A1
UnifiedClassLoader3#a45bc1
A2
UnifiedClassLoader3#a45bc1
A3
UnifiedClassLoader3#a45bc1
A4
JBossRMIClassLoader#bc450a
A5
JBossRMIClassLoader#bc450a
A6
JBossRMIClassLoader#bc450a
B1
UnifiedClassLoader3#a45bc2
B2
UnifiedClassLoader3#a45bc2
B3
UnifiedClassLoader3#a45bc2
B4
JBossRMIClassLoader#bc450b
B5
JBossRMIClassLoader#bc450b
B6
JBossRMIClassLoader#bc450b
C1
UnifiedClassLoader3#a45bc3
C2
UnifiedClassLoader3#a45bc3
C3
UnifiedClassLoader3#a45bc3
C4
JBossRMIClassLoader#bc450c
C5
JBossRMIClassLoader#bc450c
C6
JBossRMIClassLoader#bc450c
file2.log (Similar patterns except the "C" set repeats the "A" set)
A1
UnifiedClassLoader3#d0357a
A2
UnifiedClassLoader3#d0357a
A3
UnifiedClassLoader3#d0357a
A4
JBossRMIClassLoader#191777e
A5
JBossRMIClassLoader#191777e
A6
JBossRMIClassLoader#191777e
B1
UnifiedClassLoader3#d0357b
B2
UnifiedClassLoader3#d0357b
B3
UnifiedClassLoader3#d0357b
B4
JBossRMIClassLoader#191777f
B5
JBossRMIClassLoader#191777f
B6
JBossRMIClassLoader#191777f
C1
UnifiedClassLoader3#d0357a
C2
UnifiedClassLoader3#d0357a
C3
UnifiedClassLoader3#d0357a
C4
JBossRMIClassLoader#191777e
C5
JBossRMIClassLoader#191777e
C6
JBossRMIClassLoader#191777e
And after processing you get file1_processed.log
A1
UnifiedClassLoader#1
A2
UnifiedClassLoader#1
A3
UnifiedClassLoader#1
A4
JBossRMIClassLoader#2
A5
JBossRMIClassLoader#2
A6
JBossRMIClassLoader#2
B1
UnifiedClassLoader#3
B2
UnifiedClassLoader#3
B3
UnifiedClassLoader#3
B4
JBossRMIClassLoader#4
B5
JBossRMIClassLoader#4
B6
JBossRMIClassLoader#4
C1
UnifiedClassLoader#5
C2
UnifiedClassLoader#5
C3
UnifiedClassLoader#5
C4
JBossRMIClassLoader#6
C5
JBossRMIClassLoader#6
C6
and file2_processed.log
A1
UnifiedClassLoader#1
A2
UnifiedClassLoader#1
A3
UnifiedClassLoader#1
A4
JBossRMIClassLoader#2
A5
JBossRMIClassLoader#2
A6
JBossRMIClassLoader#2
B1
UnifiedClassLoader#3
B2
UnifiedClassLoader#3
B3
UnifiedClassLoader#3
B4
JBossRMIClassLoader#4
B5
JBossRMIClassLoader#4
B6
JBossRMIClassLoader#4
C1
UnifiedClassLoader#1
C2
UnifiedClassLoader#1
C3
UnifiedClassLoader#1
C4
JBossRMIClassLoader#2
C5
JBossRMIClassLoader#2
C6
JBossRMIClassLoader#2