Multiple Files merge based on file1 keys - bash

Question

I was trying to achieve merging of multiple files with main file keys.
My main file is like this
cat files.txt
Which has keys, want to compare....
1
2
3
4
5
6
7
8
9
10
11
Other inputs files like this
cat f1.txt
1 : 20
3 : 40
5 : 40
7 : 203
cat f2.txt
3 : 45
4 : 56
9 : 23
Want output like this ..
f1 f2 ....
1 20 NA
2 NA NA
3 40 45
4 56 NA
5 40 NA
6 NA NA
7 203 NA
8 NA NA
9 23 NA
10 NA NA
11 NA NA
tried this but not able to print the non-match keys
awk -F':' 'NF>1{a[$1] = a[$1]$2}END{for(i in a){print i""a[i]}}' files.txt *.txt
1 20
3 40 45
4 56
5 40
7 203
9 23
Please can someone guide me what is missing here ?

Answer 1

Complex GNU awk solution (will cover any number of files, considering system resources):
awk 'BEGIN{
PROCINFO["sorted_in"]="#ind_num_asc"; h=" ";
for(i=2;i<=ARGC;i++) h=(i==2)? h ARGV[i]: h OFS ARGV[i]; print h
}
NR==FNR{ a[$1]; next }{ b[ARGIND][$1]=$3 }
END{
for(i in a) {
printf("%d",i);
for(j in b) printf("%s%s",OFS,(i in b[j])? b[j][i] : "NA"); print ""
}
}' files.txt *.txt
An exemplary output:
f1 f2
1 20 NA
2 NA NA
3 40 45
4 NA 56
5 40 NA
6 NA NA
7 203 NA
8 NA NA
9 NA 23
10 NA NA
11 NA NA
PROCINFO["sorted_in"]="#ind_num_asc" - sorting mode (numerically in ascending order)
for(i=2;i<=ARGC;i++) h=(i==1)? h ARGV[i]: h OFS ARGV[i] - iterating through script arguments, collecting filenames.
ARGC and ARGV make the command-line arguments available to your program

Answer 2

$ cat awk-file
NR==FNR{
l=NR
next
}
NR==FNR+l{
split(FILENAME,f1,".")
a[$1]=$3
next
}
NR==FNR+l+length(a){
split(FILENAME,f2,".")
bwk -v OFS='\t' -f awk-file files.txt f1.txt f2.txt[$1]=$3
next
}
END{
print "",f1[1],f2[1]
for(i=1;i<=l;i++){
print i,(a[i]!="")?a[i]:"NR",(b[i]!="")?b[i]:"NR"
}
}
$ awk -v OFS='\t' -f awk-file files.txt f1.txt f2.txt
f1 f2
1 20 NR
2 NR NR
3 40 45
4 NR 56
5 40 NR
6 NR NR
7 203 NR
8 NR NR
9 NR 23
10 NR NR
11 NR NR
I modify the answer for your further question.
If you have 3rd, 4th files (assume to nth files), add n new blocks as followed,
NR==FNR+l+length(a)+...+length(n){
split(FILENAME,fn,".")
n[$1]=$3
}
And in your End block,
END{
print "",f1[1],f2[1],...,fn[1]
for(i=1;i<=l;i++){
print i,(a[i]!="")?a[i]:"NR",(b[i]!="")?b[i]:"NR",...,(n[i]!="")?n[i]:"NR"
}
}

Answer 3

$ cat tst.awk
ARGIND < (ARGC-1) { map[ARGIND,$1] = $NF; next }
FNR==1 {
printf "%-2s", ""
for (fileNr=1; fileNr<ARGIND; fileNr++) {
fileName = ARGV[fileNr]
sub(/\.txt$/,"",fileName)
printf "%s%s", OFS, fileName
}
print ""
}
{
printf "%-2s", $1
for (fileNr=1; fileNr<ARGIND; fileNr++) {
printf "%s%s", OFS, ((fileNr,$1) in map ? map[fileNr,$1] : "NA")
}
print ""
}
$ awk -f tst.awk f1.txt f2.txt files.txt
f1 f2
1 20 NA
2 NA NA
3 40 45
4 NA 56
5 40 NA
6 NA NA
7 203 NA
8 NA NA
9 NA 23
10 NA NA
11 NA NA
The above uses GNU awk for ARGIND, with other awks just add a line FNR==1{ARGIND++} at the start of the script.

Answer 4

Using awk and sort -n for sorting the output:
$ awk -F" *: *" '
NR==FNR {
a[$1]; next }
FNR==1 {
for(i in a)
a[i]=a[i] " NA"
h=h OFS FILENAME
}
{
match(a[$1]," NA")
a[$1]=substr(a[$1],1,RSTART-1) OFS $2 substr(a[$1],RSTART+RLENGTH)
}
END {
print h
for(i in a)
print i a[i]
}' files f1 f2 |sort -n
f1 f2
1 20 NA
2 NA NA
3 40 45
4 56 NA
5 40 NA
6 NA NA
7 203 NA
8 NA NA
9 23 NA
10 NA NA
11 NA NA
Pitfalls: 1. sort will fail with the header in certain situations. 2. Since NA is replaced with the value $2, your data can't have NA starting strings. That could probably be circumvented with replacing / NA( |$)/ but would probably cause a lot more checking in the code, so choose your NA carefully. :D
Edit:
Running it for, for example, four files:
$ awk '...' files f1 f2 f1 f2 | sort -n
1 20 20 NA NA
2 NA NA NA NA
3 40 45 40 45
4 56 56 NA NA
5 40 40 NA NA
6 NA NA NA NA
7 203 203 NA NA
8 NA NA NA NA
9 23 23 NA NA
10 NA NA NA NA
11 NA NA NA NA

Answer 5

Please use the below script to process.
FILESPATH has the list of your input files (f1.txt, f2.txt...).
INPUT has the input file (files.txt).
script.sh
FILESPATH=/home/ubuntu/work/test/
INPUT=/home/ubuntu/work/files.txt
i=0
while read line
do
FILES[ $i ]="$line"
(( i++ ))
done < <(ls $FILESPATH/*.txt)
for file in "${FILES[#]}"
do
echo -n " ${file##*/}"
done
echo ""
while IFS= read -r var
do
echo -n "$var "
for file in "${FILES[#]}"
do
VALUE=`grep "$var " $file | cut -d ' ' -f3`
if [ ! -z $VALUE ]; then
echo -n "$VALUE "
else
echo -n "NA "
fi
done
echo ""
done < "$INPUT"
========
you can use printf instead of echo to get better formatting of output.

Answer 6

This can be done via simple loop and echo statements.
#!/bin/bash
NA=" NA"
i=0
#print header module start
header[i]=" "
for file in `ls f[0-9].txt`;
do
first_part=`echo $file|cut -d. -f1`
i=$i+1
header[i]=$first_part
done
echo ${header[#]}
#print header module end
#print elements start
for element in `cat files.txt`;
do
var=$element
for file in `ls f[0-9].txt`;
do
var1=`grep -w ${element} $file`
if [[ ! -z $var1 ]] ; then
field2=`echo $var1|cut -d":" -f2`
var="$var$field2"
else
var="$var$NA"
fi
done
echo $var
done
#print elements end