Good nigt. I have this two files:
File 1 - with phenotype informations, the first column are the Ids, the orinal file has 400 rows:
ID a b c d
215 2 25 13.8354303 15.2841303
222 2 25.2 15.8507278 17.2994278
216 2 28.2 13.0482192 14.4969192
223 11 15.4 9.2714745 11.6494745
File 2 - with SNPs information, the original file has 400 lines and 42,000 characters per line.
ID t u j l
215 2 0 2 1
222 2 0 1 1
216 2 0 2 1
223 2 0 2 2
217 2 0 2 1
218 0 2 0 2
And I need to remove from file 2 individuals that do not appear in the file 1, for example:
ID t u j l
215 2 0 2 1
222 2 0 1 1
216 2 0 2 1
223 2 0 2 2
I used this code:
awk 'NR==FNR{a[$1]; next}$1 in a{print $0}' file2 file1 > file3
and I can get this output(file 3):
215 2 0 2 1
222 2 0 1 1
216 2 0 2 1
223 2 0 2 2
but I lose the header, how do I not lose the header?
To keep the header of the second file, add a condition{action} like this:
awk 'NR==FNR {a[$1]; next}
FNR==1 {print $0; next} # <= this will print the header of file2.
$1 in a {print $0}' file1 file2
NR holds the total record number while FNR is the file record number, it counts the records of the file currently being processed. Also the next statements are important, so that to continue with the next record and don't try the rest of the actions.
Related
I have two files, file1:
1 I0626_all 0 0 1 1
2 I0627_all 0 0 2 1
3 I1137_all_published 0 0 1 1
4 I1859_all 0 0 2 1
5 I2497_all 0 0 2 1
6 I2731_all 0 0 1 1
7 I4451_all 0 0 1 1
8 I0626 0 0 1 1
9 I0627 0 0 2 1
10 I0944 0 0 2 1
and file 2:
I0626_all 1 138
I0627_all 1 139
I1137_all_published 1 364
I4089 1 365
AfontovaGora2.SG 1 377
AfontovaGora3_d 1 378
At the end I want
1 I0626_all 138
2 I0627_all 139
3 I1137_all_published 364
I tried using:
awk 'NR==FNR{a[$1]=$2;next} {b[$3]} {print $1,$2,b[$3]}' file2 file1
But It doesnt work.
You may use this awk:
awk 'NR == FNR {map[$1] = $NF; next} $2 in map {print $1, $2, map[$2]}' file2 file1
1 I0626_all 138
2 I0627_all 139
3 I1137_all_published 364
I have two files that I need to merge together based on what string they contain in a specific column.
File 1 looks like this:
1 1655 1552 189
1 1433 1552 185
1 1623 1553 175
1 691 1554 182
1 1770 1554 184
1 1923 1554 182
1 1336 1554 181
1 660 1592 179
1 743 1597 179
File 2 looks like this:
1 1552 0 0 2 -9 G A A A
1 1553 0 0 2 -9 A A G A
1 1554 0 751 2 -9 A A A A
1 1592 0 577 1 -9 G A A A
1 1597 0 749 2 -9 A A G A
1 1598 0 420 1 -9 A A A A
1 1600 0 0 1 -9 A A G G
1 1604 0 1583 1 -9 A A A A
1 1605 0 1080 2 -9 G A A A
I am wanting to match column 3 from file 1 to column 2 on file 2, with my output looking like:
1 1655 1552 189 0 0 2 -9 G A A A
1 1433 1552 185 0 0 2 -9 G A A A
1 1623 1553 175 0 0 2 -9 A A G A
1 691 1554 182 0 751 2 -9 A A A A
1 1770 1554 184 0 751 2 -9 A A A A
1 1923 1554 182 0 751 2 -9 A A A A
1 1336 1554 181 0 751 2 -9 A A A A
1 660 1592 179 0 577 1 -9 G A A A
1 743 1597 179 0 749 2 -9 A A G A
I am not interested in keeping any lines in file 2 that are not in file 1. Thanks in advance!
Thanks to #Abelisto I managed to figure something out 4 hours later!
sort -k 3,3 File1.txt > Pheno1.txt
awk '($2 >0)' File2.ped > Ped1.ped
sort -k 2,2 Ped1.ped > Ped2.ped
join -1 3 -2 2 Pheno1.txt Ped2.ped > Ped3.txt
cut -d ' ' -f 1,4,5 --complement Ped3.txt > Output.ped
My real File2 actually contained negative values in the 2nd column (thankfully my real File1 didn't have any negatives) hence the use of awk to remove those rows
Using awk:
awk 'NR == FNR { arr[$2]=$3" "$4" "$5" "$6" "$7" "$8" "$9" "$10 } NR != FNR { print $1" "$2" "$3" "$4" "arr[$3] }' file2 file1
Process file2 first (NR==FNR) Set up an array called arr with the 3rd space delimited field as the index and the 3rd to 10th fields as values separated with a space. Then when processing the first file (NR!=FNR) print the 1st to the 4th space delimited fields followed by the contents of arr, index field 3.
Since $1 seems like constant 1 and I have no idea about rowcounts of either file (800,000 columns in file2 sounded a lot) I'm hashing file1 instead:
$ awk '
NR==FNR {
a[$3]=a[$3] (a[$3]==""?"":ORS) $2 OFS $3 OFS $4
next
}
($2 in a) {
n=split(a[$2],t,ORS)
for(i=1;i<=n;i++) {
$2=t[i]
print
}
}' file1 file2
Output:
1 1655 1552 189 0 0 2 -9 G A A A
1 1433 1552 185 0 0 2 -9 G A A A
1 1623 1553 175 0 0 2 -9 A A G A
1 691 1554 182 0 751 2 -9 A A A A
1 1770 1554 184 0 751 2 -9 A A A A
1 1923 1554 182 0 751 2 -9 A A A A
1 1336 1554 181 0 751 2 -9 A A A A
1 660 1592 179 0 577 1 -9 G A A A
1 743 1597 179 0 749 2 -9 A A G A
When posting a question, please add details such as row and column counts to it. Better requirements yield better answers.
These are my two imput files:
file1.txt
1 34
2 55
3 44
6 77
file2.txt
1 12
2 7
5 32
And I wish my output to be:
1 34 12
2 55 0
3 44 0
5 0 32
6 77 0
I need to do this in awk and although I was able to merge files, I do not know how to do it without losing info...
awk -F"\t" 'NR==FNR {h[$1] = $2; next }{print $1,$2,h[$2]}' file1.txt file2.txt > try.txt
awk '{ if ($3 !="") print $1,$2,$3; else print $1,$2,"0";}' try.txt > output.txt
And the output is:
1 34 12
2 55 7
3 44 0
6 77 0
Sorry, I know this must be very easy, but I am quite new in this world! Please I need help!!! Thanks in advance!!
this command gives you the desired output:
awk 'NR==FNR{a[$1]=$2;next}
{if($1 in a){print $0,a[$1];delete a[$1]}
else print $0,"0"}
END{for(x in a)print x,"0",a[x]}' file2 file1|sort -n|column -t
note that I used sort and column to sort & format the output.
output: (note I guess the 2 55 0 was a typo in your expected output)
1 34 12
2 55 7
3 44 0
5 0 32
6 77 0
Here is another way using join and awk:
join -a1 -a2 -o1.1 2.1 1.2 2.2 -e0 file1 file2 | awk '{print ($1?$1:$2),$3,$4}' OFS='\t'
1 34 12
2 55 7
3 44 0
5 0 32
6 77 0
-a switch allows to join on un-pairable lines.
-o builds our output format
-e allows to specify what should be printed for values that do not exist
awk just completes the final formatting.
I have 2 fixed length files input#1 & input#2. I want to match the rows based on the value in position 37-50 in both files (pos 37-50 will have same value in both files).
If any matching record is found then cut the value against company code & Invoice number from input file #1 (position 99 until end of line).
The cut string (from Input #1) need to be appended at the end of the record/line.
Below is the code I tried (not working) and the input files & desired output. Please provide your advice.
Code:
awk '
NR==FNR && NF>1 {
v=substr($0,37,14);
#print substr($0,37,14)
next
}
NR==FNR && ( /Company Code/ OR /Invoice Number/ ) {
sub(/Company Code/,"",$0);
sub(/Invoice Number/,"",$0);
a[v]=$0;
print $0
next
}
(substr($0,37,14) in a) {
print $0 a[substr($0,99)]
}' Input1.txt input2.txt input3.txt
End code
Input #1 beginning Start's with some white spaces
612 1111111111201402120000 2 1 111 211 Due Date 20140101
612 1111111111201402120000 2 1 111 311 Company Code 227
612 1111111111201402120000 2 1 111 411 Item Code 12
612 1111111111201402120000 2 1 111 511 Invoice Number 2014010
612 1111111111201402120000 2 2 111 611 Company Code 214
612 1111111111201402120000 2 2 111 711 Item Code 20
612 1111111111201402120000 2 2 111 811 Invoice Number 3014010
612 1111111111201402120000 2 3 111 911 Due Date 20140101
612 1111111111201402120000 2 3 111 111 Invoice Number 40140101
612 1111111111201402120000 2 3 111 121 user code 15563263636
612 1111111111201402120000 2 3 111 131 Amount Due 100000
612 111111111120140212000078978982123444 111 141 Due Date 20140101
612 111111111120140212000078978982123444 111 151 Invoice Number 50140101
612 111111111120140212000078978982123444 111 161 Amount Due 008000
Input #1 End
Input #2 beginning
input 2
510 77432201111010000 2 1 1ChK 100111000001 121000248 123456789 20111101.510.77432.20001C
510 77432201111010000 2 1 2INv 20111101.510.77432.20001D
510 77432201111010000 2 1 3INv 20111101.510.77432.20002D
510 77432201111010000 2 1 4INv 20111101.510.77432.20003D
510 77432201111010000 2 1 5INv 20111101.510.77432.20004D
510 77432201111010000 2 2 1ChK 200111000002 121000248 123456789 20111101.510.77432.20002C
510 77432201111010000 2 2 2INv 20111101.510.77432.20005D
510 77432201111010000 2 2 3INv 20111101.510.77432.20006D
510 77432201111010000 2 2 4INv 20111101.510.77432.20007D
510 77432201111010000 2 2 5INv 20111101.510.77432.20008D
510 77432201111010000 2 3 1ChK 300111000003 121000248 123456789 20111101.510.77432.20003C
510 77432201111010000 2 3 2INv 20111101.510.77432.20009D
510 77432201111010000 2 3 3INv 20111101.510.77432.20010D
510 77432201111010000 2 3 4INv 20111101.510.77432.20011D
510 77432201111010000 2 6 1ChK 600111000006 121000248 123456789 20111101.510.77432.20006C
510 77432201111010000 2 6 2INv 20111101.510.77432.20021D
510 77432201111010000 2 6 3INv 20111101.510.77432.20022D
510 77432201111010000 2 6 4INv 20111101.510.77432.20023D
510 77432201111010000 2 6 5INv 20111101.510.77432.20024D
Input #2 end
Desired outout
Desired output
510 77432201111010000 2 1 1ChK 100111000001 121000248 123456789 20111101.510.77432.20001C 2272014010 (company & Inv # from input 1)
510 77432201111010000 2 1 2INv 20111101.510.77432.20001D 2272014010
510 77432201111010000 2 1 3INv 20111101.510.77432.20002D 2272014010
510 77432201111010000 2 1 4INv 20111101.510.77432.20003D (company & Inv # from input 1)
510 77432201111010000 2 1 5INv 20111101.510.77432.20004D (company & Inv # from input 1)
510 77432201111010000 2 2 1ChK 200111000002 121000248 123456789 20111101.510.77432.20002C (company & Inv # from input 1)
510 77432201111010000 2 2 2INv 20111101.510.77432.20005D (company & Inv # from input 1)
510 77432201111010000 2 2 3INv 20111101.510.77432.20006D (company & Inv # from input 1)
510 77432201111010000 2 2 4INv 20111101.510.77432.20007D (company & Inv # from input 1)
510 77432201111010000 2 2 5INv 20111101.510.77432.20008D (company & Inv # from input 1)
510 77432201111010000 2 3 1ChK 300111000003 121000248 123456789 20111101.510.77432.20003C (company & Inv # from input 1)
510 77432201111010000 2 6 1ChK 600111000006 121000248 123456789 20111101.510.77432.20006C <there is no matching record in input 1, this will be blank>
510 77432201111010000 2 6 2INv 20111101.510.77432.20021D <there is no matching record in input 1, this will be blank>
510 77432201111010000 2 6 3INv 20111101.510.77432.20022D <there is no matching record in input 1, this will be blank>
510 77432201111010000 2 6 4INv 20111101.510.77432.20023D <there is no matching record in input 1, this will be blank>
510 77432201111010000 2 6 5INv 20111101.510.77432.20024D <there is no matching record in input 1, this will be blank>
There are several issues with your awk code.
Let's go through them step-by-step:
NR==FNR && NF>1 {...;next}NR==FNR && ... --> the next will prevent the second action from being performed for all but the first record.
NR==FNR && ( /Company Code/ OR /Invoice Number/ ) { --> OR is not a valid awk statement, the logical OR is done using || (like you use && and not AND).
print $0 a[substr($0,99)] --> a[substr($0,99)] takes everything from the 99th position of the record in your 2nd input file to look up in your array, but your key is from 37-50.
We can fix them the follwing way:
Get rid of the next in the first action and limit the 3rd action to records from the 2nd input file.
Substitute OR with ||.
Use substr($0,37,14) as key to lookup in a and substr(...,99) the result.
This results in the following code (removing your diagnostic print commands and the unused 3rd input file):
awk '
NR==FNR && NF>1 {
v=substr($0,37,14);
}
NR==FNR && ( /Company Code/ || /Invoice Number/ ) {
sub(/Company Code/,"",$0);
sub(/Invoice Number/,"",$0);
a[v]=$0;
next
}
NR!=FNR && (substr($0,37,14) in a) {
print $0 substr(a[substr($0,37,14)],99)
}' input1.txt input2.txt
Since your input was off I could not reproduce your desired output but I hope you can figure it out from here on.
Also, I shortened your code to the following version doing what I think you want it to do starting from the input given:
awk '
{key=substr($0,37,14)}
NR==FNR{
if(/Company Code/||/Invoice Number/)array[key]=substr($0,98)
next
}
(key in array){print $0,array[key]}
' input1.txt input2.txt
If you need adjustments/explanations, feel free to comment.
Try something like this (Untested):
awk '
NR==FNR && /Company Code/ {
cc[$3,$4] = $NF;
next;
}
NR==FNR && /Invoice Number/ {
inv[$3,$4] = $NF;
next;
}
NR==FNR {next}
{print $0 FS cc[$3,$4] inv[$3,$4]}' input1 input2
I wish you you all a very happy New Year.
I have a file that looks like this(example): There is no header and this file has about 10000 such rows
123 345 676 58 1
464 222 0 0 1
555 22 888 555 1
777 333 676 0 1
555 444 0 58 1
PROBLEM: I only want those rows where both field 3 and 4 have a non zero value i.e. in the above example row 1 & row 3 should be included and rest should be excluded. How can I do this?
The output should look like this:
123 345 676 58 1
555 22 888 555 1
Thanks.
awk is perfect for this kind of stuff:
awk '$3 && $4' input.txt
This will give you the output that you want.
$3 && $4 is a filter. $3 is the value of the 3rd field, $4 is the value of the forth. 0 values will be evaluated as false, anything else will be evaluated as true. If there can be negative values, than you need to write more precisely:
awk '$3 > 0 && $4 > 0' input.txt