How to merge two .txt files into one, matching each line by timestamp, using awk - bash

Question

Summary:
I currently have two .txt files imported from a survey system I am testing. Column 1 of each data file is a timestamp of the format "HHMMSS.SSSSSS". In file1, there is a second column of field intensity readings. In file2 there are two additional columns of positional information. I'm attempting to write a script that matches data points between these files by lining the timestamps up. The issue is that at no point are any of the timestamps the exact same value. The script must be able to match data points (lines in each .txt file) based the timestamp of its closest counterpart in the other file (i.e. the time 125051.354948 from file1 should "match" the nearest timestamp in file2, which is 125051.112784).
If anyone with a little bit more awk/sed/join/regex/Unix knowledge could point me in the right direction, I would be very appreciative.
What I have so far:
(Please note that the exact syntax shown here may not make sense for the sample .txt files attached in this question, there are more extensive versions of these files with more columns that were being used for testing scripts.)
I'm new to awk/Unix/shell scripting so please bear with me if some of these trial solutions don't work or don't make a whole lot of sense.
I have already attempted some solutions posted here on stack overflow using join, but it doesn't seem to want to properly sort or join either of these files:
${
join -o 1.1,2.2 -1 2 -2 1 <(sort -k 2 file1) <(sort -k 1 file2)
join -v 1 -o 1.1,1.2 -1 2 -2 1 <(sort -k 2 file1) <(sort -k 1
file2)
} | sort -k 1
Result: only outputs a similar version of the original file2
I attempted to reconfigure existing awk solutions that were posted here as well:
awk 'BEGIN {FS=OFS="\t"} NR==FNR {v[$3]=$2; next} {print $1, (v[$3] ?
v[$3] : 0)}' file1 file2 > file3
awk 'BEGIN {FS=OFS="\t"} NR==FNR {v[$1]=$2; next} {print $1, (v[$1] ?
v[$1] : 0)}' file1 file2 > file3
Result: both of these awk commands result in the output of file2's
data with nothing from file1 included (or so it seems).
awk -F '
FNR == NR {
time[$3]
next
}
{ for(i in time)
if(index($3, i) == 1) {
print
next
}
}' file1 file2 > file3
Result: keeps returning a syntax error regarding the "." of ".txt"
I looked into integrating some sort of regex or split command to the script... but was confused as to how to proceed and didn't come up with anything of substance.
Sample Data
$ cat file1.txt
125051.354948 058712.429
125052.352475 058959.934
125054.354322 058842.619
125055.352671 058772.045
125057.351794 058707.281
125058.352678 058758.959
$ cat file2.txt
125050.105886 4413.34358 07629.87620
125051.112784 4413.34369 07629.87606
125052.100811 4413.34371 07629.87605
125053.097826 4413.34373 07629.87603
125054.107361 4413.34373 07629.87605
125055.107038 4413.34375 07629.87604
125056.093783 4413.34377 07629.87602
125057.097928 4413.34378 07629.87603
125058.098475 4413.34378 07629.87606
125059.095787 4413.34376 07629.87602
Expected Result:
(Format: Column1File1 Column1File2 Column2File1 Column2File2 Column3File2)
$ cat file3.txt
125051.354948 125051.112784 058712.429 4413.34358 07629.87620
125052.352475 125052.100811 058959.934 4413.34371 07629.87605
125054.354322 125054.107361 058842.619 4413.34373 07629.87605
125055.352671 125055.107038 058772.045 4413.34375 07629.87604
125057.351794 125057.097928 058707.281 4413.34378 07629.87603
125058.352678 125058.098475 058758.959 4413.34378 07629.87606
As shown, not every data point from each file will find a match. Only pairs of lines that have the most proximal timestamps to one another will be written over to the new file
As previously mentioned, current solutions result in file3 being entirely blank, or just containing information from one of the two files (but not both)

Answer 1

Please try the following:
awk '
# find the closest element in "a" to val and return the index
function binsearch(a, val, len,
low, high, mid) {
if (val < a[1])
return 1
if (val > a[len])
return len
low = 1
high = len
while (low <= high) {
mid = int((low + high) / 2)
if (val < a[mid])
high = mid - 1
else if (val > a[mid])
low = mid + 1
else
return mid
}
return (val - a[low]) < (a[high] - val) ? high : low
}
NR == FNR {
time[FNR] = $1
position[FNR] = $2
intensity[FNR] = $3
len++
next
}
{
i = binsearch(time, $1, len)
print $1 " " time[i] " " $2 " " position[i] " " intensity[i]
}
' file2.txt file1.txt
Result:
125051.354948 125051.112784 058712.429 4413.34369 07629.87606
125052.352475 125052.100811 058959.934 4413.34371 07629.87605
125054.354322 125054.107361 058842.619 4413.34373 07629.87605
125055.352671 125055.107038 058772.045 4413.34375 07629.87604
125057.351794 125057.097928 058707.281 4413.34378 07629.87603
125058.352678 125058.098475 058758.959 4413.34378 07629.87606
Note that the 4th and 5th values in your expected result may be wrongly copy-and-pasted.
[How it works]
The key is the binsearch function which finds the closest value in the
array and returns the index to the array. I would not mention about
the algorithm in detail because it is a common "binary search" technique.

Answer 2

#!/bin/bash
if [[ $# -lt 2 ]]; then
echo "wrong args, it should be $0 file1 file2"
exit 0
fi
# clear blanks, add an extra column 'm' to file1, merge file1, file2, sort
{ awk 'NF{print $0, "m"}' "$1" ; awk 'NF' "$2"; } | sort -nk1,1 | \
\
awk '# record lines and fields in to a
{a[NR] = $0; a[NR,1] = $1; a[NR,2] = $2; a[NR,3] = $3}
END{
for(i=1; i<= NR; ++i){
# 3rd filed of file1 is "m"
if(a[i, 3] == "m"){
# get difference of column1 between current record ,previous record, next record
prevDiff = (i-1) in a && a[i-1,3] == "m" ? -1 : a[i,1] - a[i-1,1]
nextDiff = (i+1) in a && a[i+1,3] == "m" ? -1 : a[i+1,1] - a[i,1]
# compare differences, choose the close one and print.
if(prevDiff !=-1 && (nextVal == -1 || prevDiff < nextDiff))
print a[i,1], a[i-1, 1], a[i, 2], a[i-1, 2], a[i-1, 3]
else if(nextDiff !=-1 && (prevDiff == -1 || nextDiff < prevDiff))
print a[i,1], a[i+1, 1], a[i, 2], a[i+1, 2], a[i+1, 3]
else
print a[i]
}
}
}'
Out put of { awk 'NF{print $0, "m"}' "$1" ; awk 'NF' "$2"; } | sort -nk1,1 is:
125050.105886 4413.34358 07629.87620
125051.112784 4413.34369 07629.87606
125051.354948 058712.429 m
125052.100811 4413.34371 07629.87605
125052.352475 058959.934 m
125053.097826 4413.34373 07629.87603
125054.107361 4413.34373 07629.87605
125054.354322 058842.619 m
125055.107038 4413.34375 07629.87604
125055.352671 058772.045 m
125056.093783 4413.34377 07629.87602
125057.097928 4413.34378 07629.87603
125057.351794 058707.281 m
125058.098475 4413.34378 07629.87606
125058.352678 058758.959 m
125059.095787 4413.34376 07629.87602