I need to make a table of numbers, where these numbers were obtained from different files, my code is
#!/bin/sh
for K in 1.7e-2; do
dir0=Kn_${K};
for P in 1.4365 2.904; do
dir1=P${P};
for r in 0.30 0.35; do
dir2=${r};
awk '/result is =/{print $NF}' ./First/${dir0}/${dir1}/R\=${dir2}/Results.dat
done;
done;
done;
exit;
I obtain as
1
2
3
4
but I need
1 3
2 4
I was reading some posts on this topic, but these topic are about on files and not that the data were generated.
Thanks for your help and support
I obtained the data as,
1
2
3
4
5
6
7
8
Wit the pipe pr -4ts $ '\ t' (thanks #karakfa)
I obtained
1 3 5 7
2 4 6 8
then with a script to transpose as,
#!/bin/sh
awk '
{
for (i=1; i<=NF; i++) {
a[NR,i] = $i
}
}
NF>p { p = NF }
END {
for (j =1; j<=p; j++) {
str=a[1,j]
for(i=2; i<=NR; i++){
str=str" "a[i,j];
}
print str
}
}' $1
I obtained
1 2
3 4
5 6
7 8
I have a problem with interspersed the numbers,
I need
1 5
2 6
3 7
4 8
But I don't know, what is the problem with command pr and its options?
Thanks for your help
Related
I have many files with three columns in a form of:
file1 | file2
1 0 1 | 1 0 2
2 3 3 | 2 3 7
3 6 2 | 3 6 0
4 1 0 | 4 1 3
5 2 4 | 5 2 1
First two columns are the same in each file. I want to calculate a sum of 3 columns from every file to receive something like this:
1 0 3
2 3 10
3 6 2
4 1 3
5 2 5
For two files awk 'FNR==NR { _a[FNR]=$3;} NR!=FNR { $3 += _a[FNR]; print; }' file*
work perfectly (I found this solution via google). How to change it on many files?
All you need is:
awk '{sum[FNR]+=$3} ARGIND==(ARGC-1){print $1, $2, sum[FNR]}' file*
The above used GNU awk for ARGIND. With other awks just add FNR==1{ARGIND++} at the start.
Since the first two columns are same in each file:
awk 'NR==FNR{b[FNR]=$1 FS $2;}{a[FNR]+=$3}END{for(i=1;i<=length(a);i++){print b[i] FS a[i];}}' file*
Array a is used to have the cumulative sum of the 3rd column of all files.
Array b is used to the 1st and 2nd column values
In the end, we print the contents of array a and b
file1
$ cat f1
1 0 1
2 3 3
3 6 2
4 1 0
5 2 4
file2
$ cat f2
1 0 2
2 3 7
3 6 0
4 1 3
5 2 1
Output
$ awk -v start=3 'NF{for(i=1; i<=NF; i++)a[FNR, i] = i>=start ? a[FNR, i]+$i : $i }END{ for(j=1; j<=FNR; j++){ s = ""; for(i=1; i<=NF; i++){ s = (s ? s OFS:"")((j,i) in a ? a[j,i] : "") } print s } }' f1 f2
1 0 3
2 3 10
3 6 2
4 1 3
5 2 5
Better Readable
variable start decides from which column start summing, suppose if you set 2 it will start summing from column2, column3 ...and so on, from all files, since you have equal no of fields and rows, it works well
awk -v start=3 '
NF{
for(i=1; i<=NF; i++)
a[FNR, i] = i>=start ? a[FNR, i]+$i : $i
}
END{
for(j=1; j<=FNR; j++)
{
s = "";
for(i=1; i<=NF; i++)
{
s = (s ? s OFS:"")((j,i) in a ? a[j,i] : "")
}
print s
}
}
' f1 f2
I want to calculate the average of 15 files:- ifile1.txt, ifile2.txt, ....., ifile15.txt. Number of columns and rows of each file are same. Part of the data looks as
ifile1.txt ifile2.txt ifile3.txt
3 5 2 2 . 1 2 1 3 . 4 3 4 1 .
1 4 2 1 . 1 3 0 2 . 5 3 1 5 .
4 6 5 2 . 2 5 5 1 . 3 4 3 1 .
5 5 7 1 . 0 0 1 1 . 4 3 4 0 .
. . . . . . . . . . . . . . .
I would like to find over a new file which will show the average of these 15 fils.
ofile.txt
2.66 3.33 2.33 2 . (i.e. average of 3 1 4, average of 5 2 3 and so on)
2.33 3.33 1 2.66 .
3 5 4.33 1.33 .
3 2.33 4 0.66 .
. . . . .
I was trying with following, but getting error
awk'{for (i=1; i<=NF; i++)} rows=FNR;cols=NF} END
{for (i=1; i<=rows; i++){for (j=1; j<=cols; j++)
s+=$i;print $0,s/NF;s=0}}' ifile* > ofile.txt
As written:
awk'{for (i=1; i<=NF; i++)} rows=FNR;cols=NF} END
…
you get 'command not found' as the error because you must leave a space between awk and the script inside the quotes. When you fix that, you start getting into problems because there are two } and only one { on the first line of the script.
When you get around to tackling the problem, you're going to need a 2D array, indexed by line number and column number, summing the values from the files. You'll also need to know the number of files processed, and the number of columns. You can then arrange to iterate over the 2D array in the END block.
awk 'FNR == 1 { nfiles++; ncols = NF }
{ for (i = 1; i < NF; i++) sum[FNR,i] += $i
if (FNR > maxnr) maxnr = FNR
}
END {
for (line = 1; line <= maxnr; line++)
{
for (col = 1; col < ncols; col++)
printf " %f", sum[line,col]/nfiles;
printf "\n"
}
}' ifile*.txt
Given the three data files from the question:
ifile1.txt
3 5 2 2
1 4 2 1
4 6 5 2
5 5 7 1
ifile2.txt
1 2 1 3
1 3 0 2
2 5 5 1
0 0 1 1
ifile3.txt
4 3 4 1
5 3 1 5
3 4 3 1
4 3 4 0
The script I showed produces:
2.666667 3.333333 2.333333
2.333333 3.333333 1.000000
3.000000 5.000000 4.333333
3.000000 2.666667 4.000000
If you want to control the number of decimal places to 2, then use %.2f in place of %f.
$ { head -n1 ifile1.txt; paste ifile*.txt;} | awk 'NR==1{d=NF; next;} {for (i=1;i<=d;i++) {s=0; for (j=i;j<=NF;j+=d) s+=$j; printf "%.2f%s",s/(NF/d),j==NF+d?"\n":"\t";}}'
2.67 3.33 2.33 2.00
2.33 3.33 1.00 2.67
3.00 5.00 4.33 1.33
3.00 2.67 4.00 0.67
This script computes each row and prints the results before moving on to the next row. Because of this, the script does not need to hold all the data in memory at once. This is important if the data files are large.
How it works
{ head -n1 ifile1.txt; paste ifile*.txt;}
This prints just the first line of ifile1.txt. Then, the paste command causes it to print the first row of all files merged, then the second row merged, and so on:
$ paste ifile*.txt
3 5 2 2 1 2 1 3 4 3 4 1
1 4 2 1 1 3 0 2 5 3 1 5
4 6 5 2 2 5 5 1 3 4 3 1
5 5 7 1 0 0 1 1 4 3 4 0
|
The pipe symbol causes the output of the above commands to be sent as input to awk. Addressing each of the awk commands in turn:
NR==1{d=NF; next;}
For the first row, we save the number of columns in variable d. Then, we skip the rest of the commands and start over on the next line of input.
for (i=1;i<=d;i++) {s=0; for (j=i;j<=NF;j+=d) s+=$j; printf "%.2f%s",s/(NF/d),j==NF+d?"\n":"\t";}
This adds up the numbers from the respective files and prints the average.
As a multiline script:
{
head -n1 ifile1.txt
paste ifile*.txt
} |
awk '
NR==1 {d=NF; next;}
{
for (i=1;i<=d;i++)
{
s=0; for (j=i;j<=NF;j+=d)
s+=$j;
printf "%.2f%s",s/(NF/d),j==NF+d?"\n":"\t";
}
}
You need to save the sum the fields into an array when you're reading the original files. You can't access $0 and i in the END block, since there's no input line then.
awk '{rows=FNR; cols=NF; for (i = 1; i <= NF; i++) { total[FNR, i] += $i }}
FILENAME != lastfn { count++; lastfn = FILENAME }
END { for (i = 1; i <= rows; i++) {
for (j = 1; j <= cols; j++) {
printf("%s ", total[i, j]/count)
}
printf("\n")
}
}' ifile* > ofile.txt
I want to compare consecutive columns of a file and return the number of matched elements. I would prefer to use shell scripting or awk. Here is a sample bash/AWK script that I am trying to use.
#!/bin/bash
for i in 3 4 5 6 7 8 9
do
for j in 3 4 5 6 7 8 9
do
`awk "$i == $j" phased.txt | wc -l`
done
done
I have a file of size 147189*828 and I want to compare each columns and return the number of matched elements in a 828*828 matrix(A similarity matrix).
This would be fairly easy in MATLAB, but, it takes a long time to load huge files. I can compare two columns and return the number of matched elements with the following awk command:
awk '$3==$4' phased.txt | wc -l
but would need some help to do it for the entire file.
A snippet of the data that I'm working on is:
# sampleID HGDP00511 HGDP00511 HGDP00512 HGDP00512 HGDP00513 HGDP00513
M rs4124251 0 0 A G 0 A
M rs6650104 0 A C T 0 0
M rs12184279 0 0 G A T 0
................................................................................
After comparing I will compute a 6*6 matrix in this case: containing the matching percentage of these columns.
In bash, variables need a $ to be interpreted, so your awk "$i == $j" phased.txt | wc -l will be evaluated as awk "3 == 4" phased.txt | wc -l; then, because of your backticks (`), the shell will try to execute that as a command. To get awk to see $3 == $4, you need to add \$: awk "\$$i == \$$j" phased.txt | wc -l.
#!/bin/bash
for i in 3 4 5 6 7 8 9
do
for j in 3 4 5 6 7 8 9
do
awk "\$$i == \$$j" phased.txt | wc -l
done
done
Though you'll probably want to show which combination you're evaluating:
#!/bin/bash
for i in 3 4 5 6 7 8 9
do
for j in 3 4 5 6 7 8 9
do
echo "$i $j: $(awk "\$$i == \$$j" phased.txt | wc -l)"
done
done
You could actually just do the count in awk directly
#!/bin/bash
for i in 3 4 5 6 7 8 9
do
for j in 3 4 5 6 7 8 9
do
echo "$i $j: $(awk "\$$i == \$$j {count++}; END{print count}" phased.txt)"
done
done
Finally, you could just do the whole thing in awk; it'll almost certainly be faster but to be honest it's not that much cleaner: [UNTESTED]
#!/usr/bin/env awk -f
{
for (i = 3; i <= 9; i++) {
for (j = 3; j <= 9; j++) {
if ($i == $j) {
counts[i, j]++
}
}
}
}
END {
for (i = 3; i <= 9; i++) {
for (j = 3; j <= 9; j++) {
printf "%d = %d: %d\n", i, j, counts[i, j]
}
}
}
I have the following shell script that reads in data from a file inputted at the command line. The file is a matrix of numbers, and I need to separate the file by columns and then sort the columns. Right now I can read the file and output the individual columns but I am getting lost on how to sort. I have inputted a sort statement, but it only sorts the first column.
EDIT:
I have decided to take another route and actual transpose the matrix to turn the columns into rows. Since I have to later calculate the mean and median and have already successfully done this for the file row-wise earlier in the script - it was suggested to me to try and "spin" the matrix if you will to turn the columns into rows.
Here is my UPDATED code
declare -a col=( )
read -a line < "$1"
numCols=${#line[#]} # save number of columns
index=0
while read -a line ; do
for (( colCount=0; colCount<${#line[#]}; colCount++ )); do
col[$index]=${line[$colCount]}
((index++))
done
done < "$1"
for (( width = 0; width < numCols; width++ )); do
for (( colCount = width; colCount < ${#col[#]}; colCount += numCols ) ); do
printf "%s\t" ${col[$colCount]}
done
printf "\n"
done
This gives me the following output:
1 9 6 3 3 6
1 3 7 6 4 4
1 4 8 8 2 4
1 5 9 9 1 7
1 5 7 1 4 7
Though I'm now looking for:
1 3 3 6 6 9
1 3 4 4 6 7
1 2 4 4 8 8
1 1 5 7 9 9
1 1 4 5 7 7
To try and sort the data, I have tried the following:
sortCol=${col[$colCount]}
eval col[$colCount]='($(sort <<<"${'$sortCol'[*]}"))'
Also: (which is how I sorted the row after reading in from line)
sortCol=( $(printf '%s\t' "${col[$colCount]}" | sort -n) )
If you could provide any insight on this, it would be greatly appreciated!
Note, as mentioned in the comments, a pure bash solution isn't pretty. There are a number of ways to do it, but this is probably the most straight forward. The following requires reading all values per line into the array, and saving the matrix stride so it can be transposed to read all column values into a row matrix and sorted. All sorted columns are inserted into new row matrix a2. Transposing that row matrix yields your original matrix back in column sort order.
Note this will work for any rank of column matrix in your file.
#!/bin/bash
test -z "$1" && { ## validate number of input
printf "insufficient input. usage: %s <filename>\n" "${0//*\//}"
exit 1;
}
test -r "$1" || { ## validate file was readable
printf "error: file not readable '%s'. usage: %s <filename>\n" "$1" "${0//*\//}"
exit 1;
}
## function: my sort integer array - accepts array and returns sorted array
## Usage: array=( "$(msia ${array[#]})" )
msia() {
local a=( "$#" )
local sz=${#a[#]}
local _tmp
[[ $sz -lt 2 ]] && { echo "Warning: array not passed to fxn 'msia'"; return 1; }
for((i=0;i<$sz;i++)); do
for((j=$((sz-1));j>i;j--)); do
[[ ${a[$i]} -gt ${a[$j]} ]] && {
_tmp=${a[$i]}
a[$i]=${a[$j]}
a[$j]=$_tmp
}
done
done
echo ${a[#]}
unset _tmp
unset sz
return 0
}
declare -a a1 ## declare arrays and matrix variables
declare -a a2
declare -i cnt=0
declare -i stride=0
declare -i sz=0
while read line; do ## read all lines into array
a1+=( $line );
(( cnt == 0 )) && stride=${#a1[#]} ## calculate matrix stride
(( cnt++ ))
done < "$1"
sz=${#a1[#]} ## calculate matrix size
## print original array
printf "\noriginal array:\n\n"
for ((i = 0; i < sz; i += stride)); do
for ((j = 0; j < stride; j++)); do
printf " %s" ${a1[i+j]}
done
printf "\n"
done
## sort columns from stride array
for ((j = 0; j < stride; j++)); do
for ((i = 0; i < sz; i += stride)); do
arow+=( ${a1[i+j]} )
done
a2+=( $(msia ${arow[#]}) ) ## create sorted array
unset arow
done
## print the sorted array
printf "\nsorted array:\n\n"
for ((j = 0; j < cnt; j++)); do
for ((i = 0; i < sz; i += cnt)); do
printf " %s" ${a2[i+j]}
done
printf "\n"
done
exit 0
Output
$ bash sort_cols2.sh dat/matrix.txt
original array:
1 1 1 1 1
9 3 4 5 5
6 7 8 9 7
3 6 8 9 1
3 4 2 1 4
6 4 4 7 7
sorted array:
1 1 1 1 1
3 3 2 1 1
3 4 4 5 4
6 4 4 7 5
6 6 8 9 7
9 7 8 9 7
Awk script
awk '
{for(i=1;i<=NF;i++)a[i]=a[i]" "$i} #Add to column array
END{
for(i=1;i<=NF;i++){
split(a[i],b) #Split column
x=asort(b) #sort column
for(j=1;j<=x;j++){ #loop through sort
d[j]=d[j](d[j]~/./?" ":"")b[j] #Recreate lines
}
}
for(i=1;i<=NR;i++)print d[i] #Print lines
}' file
Output
1 1 1 1 1
3 3 2 1 1
3 4 4 5 4
6 4 4 7 5
6 6 8 9 7
9 7 8 9 7
Here's my entry in this little exercise. Should handle an arbitrary number of columns. I assume they're space-separated:
#!/bin/bash
linenumber=0
while read line; do
i=0
# Create an array for each column.
for number in $line; do
[ $linenumber == 0 ] && eval "array$i=()"
eval "array$i+=($number)"
(( i++ ))
done
(( linenumber++ ))
done <$1
IFS=$'\n'
# Sort each column
for j in $(seq 0 $i ); do
thisarray=array$j
eval array$j='($(sort <<<"${'$thisarray'[*]}"))'
done
# Print each array's 0'th entry, then 1, then 2, etc...
for k in $(seq 0 ${#array0[#]}); do
for j in $(seq 0 $i ); do
eval 'printf ${array'$j'['$k']}" "'
done
echo ""
done
Not bash but i think this python code worths a look showing how this task can be achieved using built-in functions.
From the interpreter:
$ cat matrix.txt
1 1 1 1 1
9 3 4 5 5
6 7 8 9 7
3 6 8 9 1
3 4 2 1 4
6 4 4 7 7
$ python
Python 2.7.3 (default, Jun 19 2012, 17:11:17)
[GCC 4.4.3] on hp-ux11
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> f = open('./matrix.txt')
>>> for row in zip(*[sorted(list(a))
for a in zip(*[a.split() for a in f.readlines()])]):
... print ' '.join(row)
...
1 1 1 1 1
3 3 2 1 1
3 4 4 5 4
6 4 4 7 5
6 6 8 9 7
9 7 8 9 7
I have a text file like this:
1 abc 2
1 rgt 2
1 yhj 2
3 gfk 4
5 kji 6
3 plo 4
3 vbn 4
5 olk 6
I want to group the rows on the basis of first and second column like this:
1 abc,rgt,yhj 2
3 gfk,plo,ybn 4
5 kji,olk 6
such that I can see what are the values of col2 for a particular pair of col1, col3.
How can I do this using shell script?
This should do it :
awk -F " " '{ a[$1" "$3]=a[$1" "$3]$2","; }END{ for (i in a)print i, a[i]; }' file.txt | sed 's/,$//g' | awk -F " " '{ tmp=$3;$3=$2;$2=tmp;print }' |sort
Just using awk:
#!/usr/bin/env awk -f
{
k = $1 "\x1C" $3
if (k in a2) {
a2[k] = a2[k] "," $2
} else {
a1[k] = $1
a2[k] = $2
a3[k] = $3
b[++i] = k
}
}
END {
for (j = 1; j <= i; ++j) {
k = b[j]
print a1[k], a2[k], a3[k]
}
}
One line:
awk '{k=$1"\x1C"$3;if(k in a2){a2[k]=a2[k]","$2}else{a1[k]=$1;a2[k]=$2;a3[k]=$3;b[++i]=k}}END{for(j=1;j<=i;++j){k=b[j];print a1[k],a2[k],a3[k]}}' file
Output:
1 abc,rgt,yhj 2
3 gfk,plo,vbn 4
5 kji,olk 6