Quickly generating test data (UUIDs, large random numbers, etc) with bash scripting - bash

Question

I have a small bash script with a function containing a case statement which echoes random data if the 1st argument matches the case parameter.
Code is as follows:
#!/usr/bin/env bash '
AC='auto-increment'
UUID='uuid'
LAT='lat'
LONG='long'
IP='ip'
generate_mock_data() {
# ARGS: $1 - data type, $2 - loop index
case ${1} in
${AC})
echo ${2} ;;
${UUID})
uuidgen ;;
${LAT})
echo $((RANDOM % 180 - 90)).$(shuf -i1000000-9999999 -n1) ;;
${LONG})
echo $((RANDOM % 360 - 180)).$(shuf -i1000000-9999999 -n1) ;;
${IP})
echo $((RANDOM%256)).$((RANDOM%256)).$((RANDOM%256)).$((RANDOM%256)) ;;
esac
}
# Writing data to file
headers=('auto-increment' 'uuid' 'lat' 'long' 'ip')
for i in {1..2500}; do
for header in "${headers[#]}"; do
echo -n $(generate_mock_data ${header} ${i}),
done
echo # New line
done >> file.csv
However, execution time is incredibly slow for just 2500 rows:
real 0m8.876s
user 0m0.576s
sys 0m0.868s
What am I doing wrong ? is there anything I can do to speed up the process ? or is bash not the right language for these type of operations ?
I also tried profiling the entire script but after looking at the logs I didn't notice any significant bottlenecks.

Answer 1

It seems you can generate a UUID pretty fast with Python, so if you just execute Python once to generate 2,500 UUIDs, and you aren't a Python programmer -like me ;-) then you can patch them up with awk:
python -c 'import uuid; print("\n".join([str(uuid.uuid4()).upper() for x in range(2500)]))' |
awk '{
lat=-90+180*rand();
lon=-180+360*rand();
ip=int(256*rand()) "." int(256*rand()) "." int(256*rand()) "." int(256*rand());
print NR,$0,lat,lon,ip
}' OFS=,
This takes 0.06s on my iMac.
OFS is the "Output Field Separator"
NR is the line number
$0 means "the whole input line"
You can try the Python on its own, like this:
python -c 'import uuid; print("\n".join([str(uuid.uuid4()).upper() for x in range(2500)]))'

Answer 2

Is Shell The Right Tool?
Not really, but if you avoid bad practices, you can make something relatively fast.
With ksh93, the below reliably runs in 0.5-0.6s wall-clock; with bash, 1.2-1.3s.
What Does It Look Like?
#!/usr/bin/env bash
# Comment these two lines if running with ksh93, obviously. :)
[ -z "$BASH_VERSION" ] && { echo "This requires bash 4.1 or newer" >&2; exit 1; }
[[ $BASH_VERSION = [123].* ]] && { echo "This requires bash 4.1 or newer" >&2; exit 1; }
uuid_stream() {
python -c '
import uuid
try:
while True:
print str(uuid.uuid4()).upper()
except IOError:
pass # probably an EPIPE because we were closed.
'
}
# generate a file descriptor that emits a shuffled stream of integers
exec {large_int_fd}< <(while shuf -r -i1000000-9999999; do :; done)
# generate a file descriptor that emits an endless stream of UUIDs
exec {uuid_fd}< <(uuid_stream)
generate_mock_data() {
typeset val
case $1 in
auto-increment) val="$2" ;;
uuid) IFS= read -r val <&"$uuid_fd" || exit;;
lat) IFS= read -r val <&"$large_int_fd" || exit
val="$((RANDOM % 180 - 90)).$val" ;;
long) IFS= read -r val <&"$large_int_fd" || exit
val="$((RANDOM % 360 - 180)).$val" ;;
ip) val="$((RANDOM%256)).$((RANDOM%256)).$((RANDOM%256)).$((RANDOM%256))" ;;
esac
printf '%s' "$val"
}
for ((i=0; i<2500; i++)); do
for header in auto-increment uuid lat long ip; do
generate_mock_data "$header" "$i"
printf ,
done
echo
done > file.csv
What's Different?
There are no command substitutions inside the inner loop. That means we don't ever use $() or any synonym for same. Each of these involves a fork() -- creating a new OS-level copy of the process -- and a wait(), with a bunch of FIFO magic to capture our output.
There are no external commands inside the inner loop. Any external command is even worse than a command substitution: They require a fork, and then additionally require an execve, with the dynamic linker and loader being invoked to pull in all the library dependencies for whichever external command is being run.
Because we don't have a command substitution stripping newlines, we have the function just not emitting them.