How to speed up reading txt file in bash script - bash

Question

I am buliding a script that reads 24 daily hourly temperature data to extract a latitude-longitude region for a smaller domain. There are three columns in each data file temperature-longitude-latitude and 188426 rows.
> ==> 20120810234500.txt <==
> 0.0362,-12.5000,33.5000
> -0.0188,-12.5000,33.5400
> -0.0732,-12.5000,33.5800
> -0.1263,-12.5000,33.6200
> -0.1778,-12.5000,33.6600
> -0.2278,-12.5000,33.7000
> -0.2761,-12.5000,33.7400
> -0.3226,-12.5000,33.7800
> -0.3677,-12.5000,33.8200
> -0.4115,-12.5000,33.8600
I have used for and while loops and awk command to read data but it takes a too long time (at least for me) to read, extract and grab the new smaller file. Here you can see the relevant part of the script
# Start 24 hours loop
lom1=-3
lom2=3
lam1=35
lam2=42
nhoras=24
n=1
while [ $n -le $nhoras ]
do
# File name (nom_file) and length (nstation=188426)
nom_file=`awk -v i=$n 'BEGIN { FS = ","} NR==i { print $1 }' lista_datos.txt`
nstation=`awk 'END{print NR}' $nom_file`
# Original data came from windows system and has carriage returns
dos2unix -q $nom_file
# Date, time values from file name
year=`echo $nom_file | cut -c 1-4`
month=`echo $nom_file | cut -c 5-6`
day=`echo $nom_file | cut -c 7-8`
hour=`echo $nom_file | cut -c 9-14`
# Part of the string to write in the new smaller file
var1=`echo $nom_file | awk '{print substr($0,1,4) " " substr($0,5,2) " " substr($0,7,2) " " substr($0,9,6)}'`
# Read rows 65000 to 125000 to gain processing time
m=65000
#while [ $m -le $nstation ] # Bucle extración datos
while [ $m -le 125000 ] # Bucle extración datos
do
station_id=$m
elevation=1.5
lat=`awk -v i=$m 'BEGIN { FS = ","} NR==i { print $3 }' $nom_file`
lon=`awk -v i=$m 'BEGIN { FS = ","} NR==i { print $2 }' $nom_file`
# As lon/lat are floating point I use this workaround to get a smaller region
lom1=`echo $lon'>'$lon1 | bc -l`
lom2=`echo $lon'<'$lon2 | bc -l`
lam1=`echo $lat'>'$lat1 | bc -l`
lam2=`echo $lat'<'$lat2 | bc -l`
if [ $lom1 -eq 1 ] && [ $lom2 -eq 1 ];
then
if [ $lam1 -eq 1 ] && [ $lam2 -eq 1 ];
then
# Second part of the string to write in the new smaller file
var2=`awk -v i=$m -v e=$elevation 'BEGIN { FS = ","} NR==i { print "'${station_id}' " $3 " " $2 " '${elevation}' 000 " $1 " 000" }' $nom_file`
# Paste
paste <(echo "$var1") <(echo "$var2") -d ' ' >> out.txt
fi # final condición lat
fi # final condición lon
m=$(( $m + 1 ))
done # End of extracting loop
# Save results
cat cabecera-dp-s.txt out.txt > dp-s$year-$month-$day-$hour
rm out.txt
n=$(( $n + 1 ))
done # End 24 hours loop
By now it takes two hours to process a single imput file. Is there any option to speed up the process?
Thanks in advance

Answer 1

Thanks to all the comments and specially thanks to #fedorqui
With the right use for awk processing speed has dramatically increased. My first attempt processed a single file in 2 hours, now 24 files have been processed in 93 minutes. There should be room for improvement but right now is fine for me. Thanks again.
I attach the script, maybe it could be useful for someone
#!/bin/bash
# RUTAS
base=/home/meteo/PROJECTES/TERMED
dades=$base/DADES
files=$base/FILES
msg_data=$dades/MSG/Agosto
treball=$base/TREBALL
# INICIO DEL SCRIPT
cd $treball
rm *
# Header for final output
cp $files/cabecera-dp-s.txt ./
# Inicio bucle dia
for dia
in 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
do
cp $msg_data/$dia/* ./
ls 2*.txt > lista_datos.txt
awk '{print substr($0,9,6)}' lista_datos.txt > lista_horas.txt
nhoras=`awk 'END{print NR}' lista_horas.txt`
# Inicio bucle hora
n=1
while [ $n -le $nhoras ]
do
# File name and size
nom_file=`awk -v i=$n 'BEGIN { FS = ","} NR==i { print $1 }' lista_datos.txt`
nstation=`awk 'END{print NR}' $nom_file`
# avoid carriage returns
dos2unix -q $nom_file
# Date values
year=`echo $nom_file | cut -c 1-4`
month=`echo $nom_file | cut -c 5-6`
day=`echo $nom_file | cut -c 7-8`
hour=`echo $nom_file | cut -c 9-14`
# Extract region, thanks fedorqui
awk -F, '$2>=-3 && $2<=3 && $3>=35 && $3<=42' $nom_file > output-$year$month$day$hour.txt
# Parte 1 de la línea de datos RAMS
var1=`echo $nom_file | awk '{print substr($0,1,4) " " substr($0,5,2) " " substr($0,7,2) " " substr($0,9,6)}'`
# station_id, latitud, longitud, elevación y temperatura para cada punto
m=1
nstation=`awk 'END{print NR}' output-$year$month$day$hour.txt`
while [ $m -le $nstation ] # Bucle extración datos
do
station_id=$m
elevation=1.5
# Parte 2 de la línea de datos RAMS
var2=`awk -v i=$m -v e=$elevation 'BEGIN { FS = ","} NR==i { print "'${station_id}' " $3 " " $2 " '${elevation}' 000 " $1 " 000" }' output-$year$month$day$hour.txt`
# Pegamos las dos partes para construir la línea de datos
paste <(echo "$var1") <(echo "$var2") -d ' ' >> out.txt
m=$(( $m + 1 ))
done # Final bucle extracción datos
# Guardamos la salida con el formato y nombre RAMS
cat cabecera-dp-s.txt out.txt > dp-s$year-$month-$day-$hour
n=$(( $n + 1 ))
rm out.txt
done # Final bucle horas
# Borra datos para evitar conflicto con lista_horas, lista_datos
rm *txt
done # Final bucle dia