Using nested loops in bash to process huge datasets - bash

Question

I am currently working on big datasets (typically 10 Gb for each) that prevent me from using R (RStudio) and dealing with data frames as I used to.
In order to deal with a restricted amount of memory (and CPU power), I've tried Julia and Bash (Shell Script) to process those files.
My question is the following: I've concatenated my files (I have more or less 1 million individual files merged into one big file) and I would like to process those big files in this way: Let's say that I have something like:
id,latitude,longitude,value
18,1,2,100
18,1,2,200
23,3,5,132
23,3,5,144
23,3,5,150
I would like to process my file saying that for id = 18, compute the max (200), the min (100) or some other propreties then go to next id and do the same. I guess some sort of nested loop in bash would work but I'm having issues doing it in an elegant way, the answers found on the Internet so far were not really helping. I cannot process it in Julia because it's too big/heavy, that's why I'm looking for answers mostly in bash.
However, I wanted to do this because I thought it would be faster to process a huge file rather than open a file, calculate, close file and go to the next one again and again. I'm not sure at all though !
Finally, which one would be better to use? Julia or Bash? Or something else?
Thank you !

Answer 1

Julia or Bash?
If you are talking about using plain bash and not some commands that could be executed in any other shell, then the answer is obviously Julia. Plain bash is magnitudes slower than Julia.
However, I would recommend to use an existing tool instead of writing your own.
GNU datamash could be what you need. You can call it from bash or any other shell.
for id = 18, compute the max (200), the min (100) [...] then go to next id and do the same
With datamash you could use the following bash command
< input.csv datamash -Ht, -g 1 min 4 max 4
Which would print
GroupBy(id),min(value),max(value)
18,100,200
23,132,150

Answer 2

Loops in bash are slow, I think that Julia is a much better fit in this case. Here is what I would do:
(Ideally) convert your data into a binary format, like NetCDF or HDF5.
load a chunk of data (e.g. 100 000 rows, not all, unless all data holds into RAM) and perform min/max per id as you propose
go to the next chunk and update the min/max for every ids
Do not load all the data at once in memory if you can avoid it. For computing easy statistics like the minimum, maximum, sum, mean, standard deviation, ... this can by done.
In my opinion, the memory overhead of julia (versus bash) are probably quite small given the size of the problem.
Be sure to read the performance tips in Julia and in particular place hoot-loops inside functions and not in global scope.
https://docs.julialang.org/en/v1/manual/performance-tips/index.html
Alternatively, such operations can also be done with specific queries in a SQL database.

Answer 3

Bash is definitely not the best option. (Fortran, baby!)
Anyway, the following can be translated to any language you want.
#!/bin/bash
function postprocess(){
# Do whatever statistics you want on the arrays.
echo "id: $last_id"
echo "lats: ${lat[#]}"
echo "lons: ${lon[#]}"
echo "vals: ${val[#]}"
}
# Set dummy start variable
last_id="not a valid id"
count=0
while read line; do
id=$( echo $line | cut -d, -f1 )
# Ignore first line
[ "$id" == "id" ] && continue
# If this is a new id, post-process the old one
if [ $id -ne $last_id -a $count -ne 0 ] 2> /dev/null; then
# Do post processing of data
postprocess
# Reset counter
count=0
# Reset value arrays
unset lat
unset lon
unset val
fi
# Increment counter
(( count++ ))
# Set last_id
last_id=$id
# Get values into arrays
lat+=($( echo $line | cut -d, -f2 ))
lon+=($( echo $line | cut -d, -f3 ))
val+=($( echo $line | cut -d, -f4 ))
done < test.txt
[ $count -gt 0 ] && postprocess

Answer 4

For this kind of problem, I'd be wary of using bash, because it isn't suited to line-by-line processing. And awk is too line-oriented for this kind of job, making the code complicated.
Something like this in perl might do the job, with a loop of loops grouping lines together by their id field.
IT070137 ~/tmp $ cat foo.pl
#!/usr/bin/perl -w
use strict;
my ($id, $latitude, $longitude, $value) = read_data();
while (defined($id)) {
my $group_id = $id;
my $min = $value;
my $max = $value;
($id, $latitude, $longitude, $value) = read_data();
while (defined($id) && $id eq $group_id) {
if ($value < $min) {
$min = $value;
}
if ($value > $max) {
$max = $value;
}
($id, $latitude, $longitude, $value) = read_data();
}
print $group_id, " ", $min, " ", $max, "\n";
}
sub read_data {
my $line = <>;
if (!defined($line)) {
return (undef, undef, undef, undef);
}
chomp($line);
my ($id, $latitude, $longitude, $value) = split(/,/, $line);
return ($id, $latitude, $longitude, $value);
}
IT070137 ~/tmp $ cat foo.txt
id,latitude,longitude,value
18,1,2,100
18,1,2,200
23,3,5,132
23,3,5,144
23,3,5,150
IT070137 ~/tmp $ perl -w foo.pl foo.txt
id value value
18 100 200
23 132 150
Or if you prefer Python:
#!/usr/bin/python -tt
from __future__ import print_function
import fileinput
def main():
data = fileinput.input()
(id, lattitude, longitude, value) = read(data)
while id:
group_id = id
min = value
(id, lattitude, longitude, value) = read(data)
while id and group_id == id:
if value < min:
min = value
(id, lattitude, longitude, value) = read(data)
print(group_id, min)
def read(data):
line = data.readline()
if line == '':
return (None, None, None, None)
line = line.rstrip()
(id, lattitude, longitude, value) = line.split(',')
return (id, lattitude, longitude, value)
main()