So I am working on a large text file made up of rows and rows of numbers,
below is just an short excerpt to help with the question, but it could even be a series of random incrementing numbers with all sorts of numerical gaps between each row.
267
368
758
936
1248
1415
1739
1917
I am looking for a way to set a fixed numerical gap between every second pair of numbers starting on the 2nd and 3rd number. such as 100 whist maintaining the numerical difference within each pair but this difference could be any number.
Such that if the numerical gap was set to 100 the above example would become:
267
368
# gap of 100
468
646
# gap of 100
746
913
# gap of 100
1013
1191
would anybody know of a possible one liner to do this in terminal of a shell script. Thanks
A little bit clumsy, but for starters this would do, I guess. It reads the list of numbers from stdin (i.e. start with cat numbers.txt or the like), then pipe it into the rest.
paste - - | {
read -r x m; echo $x; echo $m
while read -r x y; do echo $((m+=100)); echo $((m+=y-x)); done
}
267
368
468
646
746
913
1013
1191
Explanation: paste - - lets us read two numbers at once, so we can read line by line (two numbers). The first pair is printed out unchanged, but the subsequent pairs only serve as the base for calculating the diffrence, which is added onto a running variable, which also increments by 100 on each iteration.
Here's a rewrite as parametrized function, and without the use of paste:
addgaps() {
read -r m; echo $m; read -r m; echo $m
while read -r x; read -r y; do echo $((m+=$1)); echo $((m+=y-x)); done;
}
cat numbers.txt | addgaps 100
I am attempting to loop through a list of integers starting out like so:
start=000
for i in $(seq -w $start 48 006);
However, when I try this code above, the loop seems to loop once and then quit.
What do I need to modify? (The leading zeroes need to stay)
Could you please try following.
start=0
diff=6
for i in $(seq $start $diff 48);
do
printf '%03d\n' $i
done
Output will be as follows.
000
006
012
018
024
030
036
042
048
Problem in OP's tried code:
I believe you have given wrong syntax in seq it should be startpoint then increment_number then endpoint eg-->(seq(start_point increment end_point)). Since you have given them wrongly thus it is printing them only once in loop.
In your attempt it is taking starting point as 0 and should run till 6 with difference of 48 which is NOT possible so it is printing only very first integer value which is fair enough.
EDIT: As per #Cyrus sir's comment adding BASH builtin solution here without using seq.
for ((i=0; i<=48; i=i+6)); do printf '%03d\n' $i; done
seq's input takes a start, increment-by, and finish.
You've reversed the increment-by with finish: seq -w $start 48 006 means start at zero, increment by 48 to finish at 6. The simple fix is seq -w $start 6 48. Note: 006 is not needed, just 6 since seq will equalize the widths of the numbers to two places.
I want to generate different random numbers in bash . I used $RANDOM , but in my output some numbers are identical.
var1=$((1+$RANDOM%48))
var2=$((1+$RANDOM%48))
var3=$((1+$RANDOM%48))
var4=$((1+$RANDOM%48))
var5=$((1+$RANDOM%48))
var6=$((1+$RANDOM%48))
it gives me 6 numbers between 1 and 48 but i need 6 DIFFERENT numbers between 1 and 48, the fact is that im really new and i dont know even how to start.
if you want 6 different pseudo-random numbers between 1-48 this is one way to make it
$ seq 48 | shuf | head -6
18
10
17
3
11
6
or directly with shuf options (as in this answer)
shuf -i 1-48 -n 6
another method would be rejection sampling. With awk
awk 'BEGIN{srand();
do {a[int(1+rand()*48)]} while (length(a)<6);
for(k in a) print k}'
8
14
16
23
24
27
here rejection is implicit, adding the same number again won't increase the array size (essentially a hash map)
to assign result to variable is possible, but the structure begs for array use, for example
declare -a randarray
readarray randarray < <(seq 48 | shuf | head -6)
you can access the individual elements with the corresponding index (0-based)
echo ${randarray[3]}
In general if your number of samples are close to the range of the sample space, you will want shuffle (extreme case if you want N numbers from the range 1-N, what you're asking is a random permutation), or if the ratio is small, rejection sampling might be better, (extreme case you want one random variable). Rejection sampling is used mostly if you have other conditions to eliminate a sample. However, direct use of shuf with options is very fast already and you may not need rejection sampling method at all for basic uses.
shuf -i 1-48 -n 6
will give you 6 random numbers between 1 and 48. All will be different per default.
This algorithm will roll six random numbers and check each of them to make sure there haven't been any duplicates.
declare -a nums
let times=6
for i in $(eval echo {0..$times}); do
let nums[$i]=0
let dup=1
while [ $dup -eq 1 ]; do
let temp=$((1+$RANDOM%48))
let dup=0
for j in $(eval echo {0..${#nums[*]}}); do
let cur=$((${nums[$j]}))
if ! [ "$cur" -eq "$cur" ] 2> /dev/null; then
break
fi
if [ "$cur" -eq "$(eval echo $temp)" ]; then
let dup=1
fi
done
done
let nums[$(($j - 1))]=$temp
done
echo ${nums[#]:0}
exit 1
var=$RANDOM creates random numbers but how can i specify a range like between 0 and 12 for instance?
If you already have your random number, you can say
var=$RANDOM
var=$[ $var % 13 ]
to get numbers from 0..12.
Edit:
If you want to produce numbers from $x to $y, you can easily modify this:
var=$[ $x + $var % ($y + 1 - $x) ]
Between 0 and 12 (included):
echo $((RANDOM % 13))
Edit: Note that this method is not strictly correct. Because 32768 is not a multiple of 13, the odds for 0 to 8 to be generated are slightly higher (0.04%) than the remaining numbers (9 to 12).
Here is shell function that should give a balanced output:
randomNumber()
{
top=32768-$((32768%($1+1)))
while true; do
r=$RANDOM
[ r -lt $top ] && break
done
echo $((r%$1))
}
Of course, something better should be designed if the higher value of the range exceed 32767.
An alternative using shuf available on linux (or coreutils to be exact):
var=$(shuf -i0-12 -n1)
Here you go
echo $(( $RANDOM % 12 ))
I hope this helps.
This document has some examples of using this like RANGE and FLOOR that might be helpful: http://tldp.org/LDP/abs/html/randomvar.html
On FreeBSD and possibly other BSDs you can use:
jot -r 3 0 12
This will create 3 random numbers from 0 to 12 inclusively.
Another option, if you only need a single random number per script, you can do:
var=$(( $$ % 13 ))
This will use the PID of the script as the seed, which should be mostly random. The range again will be from 0 to 12.
How to generate a random number within a range in Bash?
Use $RANDOM. It's often useful in combination with simple shell arithmetic. For instance, to generate a random number between 1 and 10 (inclusive):
$ echo $((1 + $RANDOM % 10))
3
The actual generator is in variables.c, the function brand(). Older versions were a simple linear generator. Version 4.0 of bash uses a generator with a citation to a 1985 paper, which presumably means it's a decent source of pseudorandom numbers. I wouldn't use it for a simulation (and certainly not for crypto), but it's probably adequate for basic scripting tasks.
If you're doing something that requires serious random numbers you can use /dev/random or /dev/urandom if they're available:
$ dd if=/dev/urandom count=4 bs=1 | od -t d
Please see $RANDOM:
$RANDOM is an internal Bash function
(not a constant) that returns a
pseudorandom integer in the range
0 - 32767. It should not be used to
generate an encryption key.
You can also use shuf (available in coreutils).
shuf -i 1-100000 -n 1
Try this from your shell:
$ od -A n -t d -N 1 /dev/urandom
Here, -t d specifies that the output format should be signed decimal; -N 1 says to read one byte from /dev/urandom.
you can also get random number from awk
awk 'BEGIN {
# seed
srand()
for (i=1;i<=1000;i++){
print int(1 + rand() * 100)
}
}'
I like this trick:
echo ${RANDOM:0:1} # random number between 1 and 9
echo ${RANDOM:0:2} # random number between 1 and 99
...
There is $RANDOM.
I don't know exactly how it works. But it works.
For testing, you can do :
echo $RANDOM
bash 5.1 introduces a new variable, SRANDOM, which gets its random data from the system's entropy engine and so is not linear and cannot be reseeded to get an identical random sequence. This variable can be used as a substitute for RANDOM for generating more random numbers.
$ echo $((1 + SRANDOM % 10))
4
Random number between 0 and 9 inclusive.
echo $((RANDOM%10))
I wrote several articles on this.
https://linuxconfig.org/generating-random-numbers-in-bash-with-examples
https://linuxconfig.org/random-entropy-in-bash
https://www.cloudsavvyit.com/7572/how-to-generate-better-random-numbers-at-the-bash-command-line/
$ RANDOM=$(date +%s%N | cut -b10-19)
$ echo $(( $RANDOM % 113 + 13 ))
The above will give a number between 13 and 125 (113-1+13), with reasonable random entropy.
If you are using a linux system you can get a random number out of /dev/random or /dev/urandom. Be carefull /dev/random will block if there are not enough random numbers available. If you need speed over randomness use /dev/urandom.
These "files" will be filled with random numbers generated by the operating system. It depends on the implementation of /dev/random on your system if you get true or pseudo random numbers. True random numbers are generated with help form noise gathered from device drivers like mouse, hard drive, network.
You can get random numbers from the file with dd
Reading from /dev/random or /dev/urandom character special files is the way to go.
These devices return truly random numbers when read and are designed
to help application software choose secure keys for encryption. Such
random numbers are extracted from an entropy pool that is contributed
by various random events. {LDD3, Jonathan Corbet, Alessandro
Rubini, and Greg Kroah-Hartman]
These two files are interface to kernel randomization, in particular
void get_random_bytes_arch(void* buf, int nbytes)
which draws truly random bytes from hardware if such function is by hardware implemented (usually is), or it draws from entropy pool (comprised of timings between events like mouse and keyboard interrupts and other interrupts that are registered with SA_SAMPLE_RANDOM).
dd if=/dev/urandom count=4 bs=1 | od -t d
This works, but writes unneeded output from dd to stdout. The command below gives just the integer I need. I can even get specified number of random bits as I need by adjustment of the bitmask given to arithmetic expansion:
me#mymachine:~/$ x=$(head -c 1 /dev/urandom > tmp && hexdump
-d tmp | head -n 1 | cut -c13-15) && echo $(( 10#$x & 127 ))
I have taken a few of these ideas and made a function that should perform quickly if lots of random numbers are required.
calling od is expensive if you need lots of random numbers. Instead I call it once and store 1024 random numbers from /dev/urandom. When rand is called, the last random number is returned and scaled. It is then removed from cache. When cache is empty, another 1024 random numbers is read.
Example:
rand 10; echo $RET
Returns a random number in RET between 0 and 9 inclusive.
declare -ia RANDCACHE
declare -i RET RAWRAND=$(( (1<<32)-1 ))
function rand(){ # pick a random number from 0 to N-1. Max N is 2^32
local -i N=$1
[[ ${#RANDCACHE[*]} -eq 0 ]] && { RANDCACHE=( $(od -An -tu4 -N1024 /dev/urandom) ); } # refill cache
RET=$(( (RANDCACHE[-1]*N+1)/RAWRAND )) # pull last random number and scale
unset RANDCACHE[${#RANDCACHE[*]}-1] # pop read random number
};
# test by generating a lot of random numbers, then effectively place them in bins and count how many are in each bin.
declare -i c; declare -ia BIN
for (( c=0; c<100000; c++ )); do
rand 10
BIN[RET]+=1 # add to bin to check distribution
done
for (( c=0; c<10; c++ )); do
printf "%d %d\n" $c ${BIN[c]}
done
UPDATE: That does not work so well for all N. It also wastes random bits if used with small N. Noting that (in this case) a 32 bit random number has enough entropy for 9 random numbers between 0 and 9 (10*9=1,000,000,000 <= 2*32) we can extract multiple random numbers from each 32 random source value.
#!/bin/bash
declare -ia RCACHE
declare -i RET # return value
declare -i ENT=2 # keep track of unused entropy as 2^(entropy)
declare -i RND=RANDOM%ENT # a store for unused entropy - start with 1 bit
declare -i BYTES=4 # size of unsigned random bytes returned by od
declare -i BITS=8*BYTES # size of random data returned by od in bits
declare -i CACHE=16 # number of random numbers to cache
declare -i MAX=2**BITS # quantum of entropy per cached random number
declare -i c
function rand(){ # pick a random number from 0 to 2^BITS-1
[[ ${#RCACHE[*]} -eq 0 ]] && { RCACHE=( $(od -An -tu$BYTES -N$CACHE /dev/urandom) ); } # refill cache - could use /dev/random if CACHE is small
RET=${RCACHE[-1]} # pull last random number and scale
unset RCACHE[${#RCACHE[*]}-1] # pop read random number
};
function randBetween(){
local -i N=$1
[[ ENT -lt N ]] && { # not enough entropy to supply ln(N)/ln(2) bits
rand; RND=RET # get more random bits
ENT=MAX # reset entropy
}
RET=RND%N # random number to return
RND=RND/N # remaining randomness
ENT=ENT/N # remaining entropy
};
declare -ia BIN
for (( c=0; c<100000; c++ )); do
randBetween 10
BIN[RET]+=1
done
for c in ${BIN[*]}; do
echo $c
done
Maybe I am a bit too late, but what about using jot to generate a random number within a range in Bash?
jot -r -p 3 1 0 1
This generates a random (-r) number with 3 decimal places precision (-p). In this particular case, you'll get one number between 0 and 1 (1 0 1). You can also print sequential data. The source of the random number, according to the manual, is:
Random numbers are obtained through arc4random(3) when no seed is specified, and through
random(3) when a seed is given.
Generate random number in the range of 0 to n (signed 16-bit integer). Result set in $RAND variable. For example:
#!/bin/bash
random()
{
local range=${1:-1}
RAND=`od -t uI -N 4 /dev/urandom | awk '{print $2}'`
let "RAND=$RAND%($range+1)"
}
n=10
while [ $(( n -=1 )) -ge "0" ]; do
random 500
echo "$RAND"
done
What about:
perl -e 'print int rand 10, "\n"; '
You can use a seed, see documentation:
RANDOM=$(date +%s%N | cut -b10-19)
echo $(( $RANDOM % 100 + 1 ))
Based on the great answers of #Nelson, #Barun and #Robert, here is a Bash script that generates random numbers.
Can generate how many digits you want.
each digit is separately generated by /dev/urandom which is much better than Bash's built-in $RANDOM
#!/usr/bin/env bash
digits=10
rand=$(od -A n -t d -N 2 /dev/urandom |tr -d ' ')
num=$((rand % 10))
while [ ${#num} -lt $digits ]; do
rand=$(od -A n -t d -N 1 /dev/urandom |tr -d ' ')
num="${num}$((rand % 10))"
done
echo $num
Random branching of a program or yes/no; 1/0; true/false output:
if [ $RANDOM -gt 16383 ]; then # 16383 = 32767/2
echo var=true/1/yes/go_hither
else
echo var=false/0/no/go_thither
fi
of if you lazy to remember 16383:
if (( RANDOM % 2 )); then
echo "yes"
else
echo "no"
fi
Wanted to use /dev/urandom without dd and od
function roll() { local modulus=${1:-6}; echo $(( 1 + 0x$(env LC_CTYPE=C tr -dc '0-9a-fA-F' < /dev/urandom | head -c5 ) % $modulus )); }
Testing
$ roll
5
$ roll 12
12
Just how random is it?
$ (echo "count roll percentage"; i=0; while [ $i -lt 10000 ]; do roll; i=$((i+1)); done | sort | uniq -c | awk '{print $0,($1/10000*100)"%"}') | column -t
count roll percentage
1625 1 16.25%
1665 2 16.65%
1646 3 16.46%
1720 4 17.2%
1694 5 16.94%
1650 6 16.5%
Generate random 3-digit number
This is great for creating sample data. Example: put all testing data in a directory called "test-create-volume-123", then after your test is done, zap the entire directory. By generating exactly three digits, you don't have weird sorting issues.
printf '%02d\n' $((1 + RANDOM % 100))
This scales down, e.g. to one digit:
printf '%01d\n' $((1 + RANDOM % 10))
It scales up, but only to four digits. See above as to why :)
Pure Bash random number without moduloing
lowerRange=10 # inclusive
upperRange=20 # exclusive
randomNumber=$(( RANDOM * ( upperRange - lowerRange) / 32767 + lowerRange ))
A bash function that uses perl to generate a random number of n digits. Specify either the number of digits or a template of n 0s.
rand() {
perl -E '$ARGV[0]||=""; $ARGV[0]=int($ARGV[0])||length($ARGV[0]); say join "", int(rand(9)+1)*($ARGV[0]?1:0), map { int(rand(10)) } (0..($ARGV[0]||0)-2)' $1
}
Usage:
$ rand 3
381
$ rand 000
728
Demonstration of calling rand n, for n between 0 and 15:
$ for n in {0..15}; do printf "%02d: %s\n" $n $(rand $n); done
00: 0
01: 3
02: 98
03: 139
04: 1712
05: 49296
06: 426697
07: 2431421
08: 82727795
09: 445682186
10: 6368501779
11: 51029574113
12: 602518591108
13: 5839716875073
14: 87572173490132
15: 546889624135868
Demonstration of calling rand n, for n a template of 0s between length 0 and 15
$ for n in {0..15}; do printf "%15s :%02d: %s\n" $(printf "%0${n}d" 0) $n $(rand $(printf "%0${n}d" 0)); done
0 :00: 0
0 :01: 0
00 :02: 70
000 :03: 201
0000 :04: 9751
00000 :05: 62237
000000 :06: 262860
0000000 :07: 1365194
00000000 :08: 83953419
000000000 :09: 838521776
0000000000 :10: 2355011586
00000000000 :11: 95040136057
000000000000 :12: 511889225898
0000000000000 :13: 7441263049018
00000000000000 :14: 11895209107156
000000000000000 :15: 863219624761093
Here is a function I wrote which will output a random number in a desired range>
Description:
random <min> <max>
Generate a random number from min to max, inclusive. Both min and max can be
positive OR negative numbers, and the generated random number can be negative too, so
long as the range (max - min + 1) is less than or equal to 32767. Max must be >= min.
The core of it is this:
random() {
min="$1"
max="$2"
range=$((max - min + 1))
rand=$((min + (RANDOM % range)))
echo "$rand"
}
Usage:
# general form: obtain a random number between min and max, inclusive
random <min> <max>
# Example: obtain a random number from -10 to 10, inclusive
random -10 10
This works from the bash built-in variable RANDOM, which probably just uses C rand() under the hood, since they both have a max value of 32767--see:
https://en.cppreference.com/w/c/numeric/random/rand
https://en.cppreference.com/w/c/numeric/random/RAND_MAX
For the bash documentation, see man bash:
RANDOM
Each time this parameter is referenced, a random integer between 0 and 32767 is generated. The sequence of random numbers may be initialized by assigning a value to RANDOM. If RANDOM is unset, it loses its special properties, even if it is subsequently reset.
Robust, runnable, sourceable version of the script
Here is a much more robust version of my random function above. It includes full error checking, bounds checking, a help menu via random --help or random -h, and a special run_check feature which allows you to source OR run this script so that you can source it to import the random function into any other script--just like you can do in Python!
random.sh <-- click this link to always get the latest version from my eRCaGuy_dotfiles repo.
RETURN_CODE_SUCCESS=0
RETURN_CODE_ERROR=1
HELP_STR="\
Generate a random integer number according to the usage styles below.
USAGE STYLES:
'random'
Generate a random number from 0 to 32767, inclusive (same as bash variable 'RANDOM').
'random <max>'
Generate a random number from 0 to 'max', inclusive.
'random <min> <max>'
Generate a random number from 'min' to 'max', inclusive. Both 'min' and 'max' can be
positive OR negative numbers, and the generated random number can be negative too, so
long as the range (max - min + 1) is less than or equal to 32767. Max must be >= min.
This file is part of eRCaGuy_dotfiles: https://github.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles
"
print_help() {
echo "$HELP_STR" | less -RFX
}
# Get a random number according to the usage styles above.
# See also `utils_rand()` in utilities.c:
# https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world/blob/master/c/utilities.c#L176
random() {
# PARSE ARGUMENTS
# help menu
if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
print_help
exit $RETURN_CODE_SUCCESS
fi
# 'random'
if [ $# -eq 0 ]; then
min=0
max="none"
# 'random max'
elif [ $# -eq 1 ]; then
min=0
max="$1"
# 'random min max'
elif [ $# -eq 2 ]; then
min="$1"
max="$2"
else
echo "ERROR: too many arguments."
exit "$RETURN_CODE_ERROR"
fi
# CHECK FOR ERRORS
if [ "$max" = "none" ]; then
rand="$RANDOM"
echo "$rand"
exit "$RETURN_CODE_SUCCESS"
fi
if [ "$max" -lt "$min" ]; then
echo "ERROR: max ($max) < min ($min). Max must be >= min."
exit "$RETURN_CODE_ERROR"
fi
# CALCULATE THE RANDOM NUMBER
# See `man bash` and search for `RANDOM`. This is a limitation of that value.
RAND_MAX=32767
range=$((max - min + 1))
if [ "$range" -gt "$RAND_MAX" ]; then
echo "ERROR: the range (max - min + 1) is too large. Max allowed = $RAND_MAX, but actual" \
"range = ($max - $min + 1) = $range."
exit "$RETURN_CODE_ERROR"
fi
# NB: `RANDOM` is a bash built-in variable. See `man bash`, and also here:
# https://stackoverflow.com/a/1195035/4561887
rand=$((min + (RANDOM % range)))
echo "$rand"
}
# Set the global variable `run` to "true" if the script is being **executed** (not sourced) and
# `main` should run, and set `run` to "false" otherwise. One might source this script but intend
# NOT to run it if they wanted to import functions from the script.
# See:
# 1. *****https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world/blob/master/bash/argument_parsing__3_advanced__gen_prog_template.sh
# 1. my answer: https://stackoverflow.com/a/70662049/4561887
# 1. https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world/blob/master/bash/check_if_sourced_or_executed.sh
run_check() {
# This is akin to `if __name__ == "__main__":` in Python.
if [ "${FUNCNAME[-1]}" == "main" ]; then
# This script is being EXECUTED, not sourced
run="true"
fi
}
# ----------------------------------------------------------------------------------------------------------------------
# Main program entry point
# ----------------------------------------------------------------------------------------------------------------------
# Only run main function if this file is being executed, NOT sourced.
run="false"
run_check
if [ "$run" == "true" ]; then
random "$#"
fi
No other dependency is needed:
$(((RANDOM % $((upperBound - lowerBound))) + lowerBound))
The random number range is [lowerBound,upperBound)