shell script runs out of memory - bash

Question

I have written the following random-number generator shell script:
for i in $(seq 1 $1) #for as many times, as the first argument ($1) defines...
do
echo "$i $((RANDOM%$2))" #print the current iteration number and a random number in [0, $2)
done
I run it like that:
./generator.sh 1000000000 101 > data.txt
to generate 1B rows of an id and a random number in [0,100] and store this data in file data.txt.
My desired output is:
1 39
2 95
3 61
4 27
5 85
6 44
7 49
8 75
9 52
10 66
...
It works fine for small number of rows, but with 1B, I get the following OOM error:
./generator.sh: xrealloc: ../bash/subst.c:5179: cannot allocate 18446744071562067968 bytes (4299137024 bytes allocated)
Which part of my program creates the error?
How could I write the data.txt file line-by-line?
I have tried replacing the echo line with:
echo "$i $((RANDOM%$2))" >> $3
where $3 is data.txt, but I see no difference.

Answer 1

The problem is your for loop:
for i in $(seq 1 $1)
This will first expand $(seq 1 $1), creating a very big list, which you then pass to for.
Using while, however, we can read the output of seq line-by-line, which will take a small amount of memory:
seq 1 1000000000 | while read i; do
echo $i
done

Answer 2

$(seq 1 $1) is computing the whole list before iterating over it. So it takes memory to store the entire list of 10^9 numbers, which is a lot.
I am not sure if you can make seq run lazily, i.e, get the next number only when needed. You can do a simple for loop instead:
for ((i=0; i<$1;++i))
do
echo "$i $((RANDOM%$2))"
done

Answer 3

If you want it fast this should work.
You will need to compile it using g++ using the form
g++ -o <executable> <C++file>
For example i did it this way
g++ -o inseq.exe CTest.cpp
CTest.cpp
#include <iostream>
#include <string>
#include <fstream>
#include <iomanip>
#include <cstdlib>
#include <sstream>
int main (int argc,char *argv[])
{
std::stringstream ss;
int x = atoi(argv[1]);
for(int i=1;i<=x;i++)
{
ss << i << "\n";
if(i%10000==0)
{
std::cout << ss.rdbuf();
ss.clear();
ss.str(std::string());
}
}
std::cout << ss.rdbuf();
ss.clear();
ss.str(std::string());
}
Speed comparisons
Lowest speeds of 3 tests for each of the methods presented for a 1000000 line file.
Jidder
$ time ./inseq 1000000 > file
real 0m0.143s
user 0m0.131s
sys 0m0.011s
Carpetsmoker
$ cat Carpet.sh
#!/bin/bash
seq 1 $1 | while read i; do
echo $i
done
.
$ time ./Carpet.sh 1000000 > file
real 0m12.223s
user 0m9.753s
sys 0m2.140s
Hari Shankar
$ cat Hari.sh
#!/bin/bash
for ((i=0; i<$1;++i))
do
echo "$i $((RANDOM%$2))"
done
.
$ time ./Hari.sh 1000000 > file
real 0m9.729s
user 0m8.084s
sys 0m1.064s
As you can see from the results, my way is slightly faster by about 60-70*.
Edit
Because python is great
$ cat Py.sh
#!/usr/bin/python
for x in xrange(1, 1000000):
print (x)
'
$ time ./Py.sh >file
real 0m0.543s
user 0m0.499s
sys 0m0.016s
4* slower than c++ so if the file was going to take an hour to make it would take 4 with these two lines.
EDIT 2
Decided to try Python and c++ on the 1000000000 line file
For a none CPU-intensive task this seems to be using a lottt of cpu
PID USER %CPU TIME+ COMMAND
56056 me 96 2:51.43 Py.sh
Results for Python
real 9m37.133s
user 8m53.550s
sys 0m8.348s
Results for c++
real 3m9.047s
user 2m53.400s
sys 0m2.842s