I'd like to apologize if my question had already been asked, but english isn't my native language and I didn't find the answer. I'd like to have a bash script that executes a program I'll call MyProgram, and I want it to run with a fixed number of arguments which consist in random numbers. I'd like to have something like this:
./MyProgram for(i = 0; i < 1000; i++) $(($RANDOM%200-100))
How should I go about this?
You (mostly) just have the loop and the actual program call inverted.
for ((i=0; i < 1000; i++)); do
./MyProgram $((RANDOM%200 - 100))
done
If, however, you actually want 1000 different arguments passed to a single call, you have to build up a list first.
args=()
for ((i=0; i < 1000; i++)); do
args+=( $((RANDOM%200 - 100)) )
done
./MyProgram "${args[#]}"
The
$RANDOM % 200 - 100
is the same as the next perl
perl -E 'say int(200*rand() -100) for (1..1000)'
e.g. the
perl -E 'say int(200*rand() -100) for (1..1000)' | xargs -n1 ./MyProgram
will run like:
./MyProgram -10
./MyProgram 13
... 1000 times ...
./MyProgram 55
./MyProgram -31
if you need 1000 args
./MyProgram $(perl -E 'say int(200*rand() -100) for (1..1000)')
will produce
./MyProgram 5 -41 -81 -79 -14 ... 1000 numbers ... -63 -9 95 -9 -29
In addition to what #chepner says, you can also use the for ... in style of for loop. This looks like:
for a in one two three; do
echo "${a}"
done
which would produce the result:
one
two
three
In other words, the list of words after the in part, separated by spaces, is looped over, with each iteration of the loop having a different word in the variable a.
To call your program 1000 times (or just modify to produce the list of arguments to run it once as in #chepner's answer) you could then do:
for a in $(seq 1 1000); do
./MyProgram $((RANDOM%200 - 100))
done
where the output of the seq command is providing the list of values to loop over. Although the traditional for loop may be more immediately obvious to many programmers, I like for ... in because it can be applied in lots of situations. A crude and mostly pointless ls, for example:
for a in *; do
echo "${a}"
done
for ... in is probably the bit of "advanced" bash that I find the most useful, and make use of it very frequently.
Related
I have a bash loop that I run to copy 2 files from the hpc to my local drive recursively over the processors and all the timesteps. On the hpc the timesteps are saved as
1 2 3
whereas the bash loop interprets it as
1.0 2.0 3.0
probably because of the 0.5 increment. Is there a way to get the $j to be changed to whole number (without the decimal) when running the script?
Script I use:
for i in $(seq 0 1 23)
do
mkdir Run1/processor$i
for j in $(seq 0 0.5 10);
do
mkdir Run1/processor$i/$j
scp -r xx#login.hpc.xx.xx:/scratch/Run1/processor$i/$j/p Run1/processor$i/$j/
scp -r xx#login.hpc.xx.xx:/scratch/Run1/processor$i/$j/U Run1/processor$i/$j/
done
done
Result:
scp: /scratch/Run1/processor0/1.0/p: No such file or directory
The correct directory that exists is
/scratch/Run1/processor0/1
Thanks!
well, yes!
but: Depending on what the end result is.
I will assume you want to floor the decimal number. I can think of 2 options:
pipe the number to cut
do a little bit of perl
for i in $(seq 0 1 23); do
for j in $(seq 0 0.5 10); do
# pipe to cut
echo /scratch/Run1/processor$i/$(echo $j | cut -f1 -d".")/U Run1/processor"$i/$j"/
# pipe to perl
echo /scratch/Run1/processor$i/$(echo $j | perl -nl -MPOSIX -e 'print floor($_);')/U Run1/processor"$i/$j"/
done
done
result:
...
/scratch/Run1/processor23/9/U Run1/processor23/9/
/scratch/Run1/processor23/9/U Run1/processor23/9.5/
/scratch/Run1/processor23/9/U Run1/processor23/9.5/
/scratch/Run1/processor23/10/U Run1/processor23/10/
/scratch/Run1/processor23/10/U Run1/processor23/10/
edit :
Experimented a litle, found another way:
echo /scratch/Run1/processor$i/${j%%.[[:digit:]]}/U Run1/processor"$i/$j"/
How to generate 9 digit random number in shell?
I am trying something like this but it only gave numbers below 32768.
#!/bin/bash
mo=$((RANDOM%999999999))
echo "********Random"$mo
Please help
output should be ********Random453351111
In Linux with /dev/urandom:
$ rnd=$(tr -cd "[:digit:]" < /dev/urandom | head -c 9) && echo $rnd
463559879
I think this should make it
shuf -i 99999999-999999999 -n 1
As a work around, we could just simply ask for 1 random integer, for n times:
rand=''
for i in {1..9}; do
rand="${rand}$(( $RANDOM % 10 ))"
done
echo $rand
Try it online!
Note [1]: Since RANDOM's upper limit has a final digit of 7, there's a slightly lesser change for the 'generated' number to contain 8 or 9's.
Because of RANDOM's limited range, it can only be used to retrieve four base-10 digits at a time. Thus, to retrieve 9 digits, you need to call it three times.
If we don't care much about performance (are willing to pay process substitution costs), this may look like:
#!/usr/bin/env bash
get4() {
local newVal=32768
while (( newVal > 29999 )); do # avoid bias because of remainder
newVal=$RANDOM
done
printf '%04d' "$((newVal % 10000))"
}
result="$(get4)$(get4)$(get4)"
result=$(( result % 1000000000 ))
printf '%09d\n' "$result"
If we do care about performance, it may instead look like:
#!/usr/bin/env bash
get4() {
local newVal=32768 outVar=$1
while (( newVal > 29999 )); do # avoid bias because of remainder
newVal=$RANDOM
done
printf -v "$outVar" '%04d' "$((newVal % 10000))"
}
get4 out1; get4 out2; get4 out3
result="${out1}${out2}${out3}"
result=$(( result % 1000000000 ))
printf '%09d\n' "$result"
Use perl, as follows :
perl -e print\ rand | cut -c 3-11
Or
perl -MPOSIX -e 'print floor rand 10**9'
I have a for loop in bash that writes values to a file. However, because there are a lot of values, the process takes a long time, which I think can be saved by improving the code.
nk=1152
nb=24
for k in $(seq 0 $((nk-1))); do
for i in $(seq 0 $((nb-1))); do
for j in $(seq 0 $((nb-1))); do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
I've moved the output action to after the entire loop is done rather than echo -e "$k\t$i\t$j" >> file.dat to avoid opening and closing the file many times. However, the speed the script writes to the file is still rather slow, ~ 10kbps.
Is there a better way to improve the IO?
Many thanks
Jacek
It looks like the seq calls are fairly punishing since that is a separate process. Try this just using shell math instead:
for ((k=0;k<=$nk-1;k++)); do
for ((i=0;i<=$nb-1;i++)); do
for ((j=0;j<=$nb-1;j++)); do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
It takes just 7.5s on my machine.
Another way is to compute the sequences just once and use them repeatedly, saving a lot of shell calls:
nk=1152
nb=24
kseq=$(seq 0 $((nk-1)))
bseq=$(seq 0 $((nb-1)))
for k in $kseq; do
for i in $bseq; do
for j in $bseq; do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
This is not really "better" than the first option, but it shows how much of the time is spent spinning up instances of seq versus actually getting stuff done.
Bash isn't always the best for this. Consider this Ruby equivalent which runs in 0.5s:
#!/usr/bin/env ruby
nk=1152
nb=24
nk.times do |k|
nb.times do |i|
nb.times do |j|
puts "%d\t%d\t%d" % [ k, i, j ]
end
end
end
What is the most time consuming is calling seq in a nested loop. Keep in mind that each time you call seq it loads command from disk, fork a process to run it, capture the output, and store the whole output sequence into memory.
Instead of calling seq you could use an arithmetic loop:
#!/usr/bin/env bash
declare -i nk=1152
declare -i nb=24
declare -i i j k
for ((k=0; k<nk; k++)); do
for (( i=0; i<nb; i++)); do
for (( j=0; j<nb; j++)); do
printf '%d\t%d\t%d\n' "$k" "$i" "$j"
done
done
done > file.dat
Running seq in a subshell consumes most of the time.
Switch to a different language that provides all the needed features without shelling out. For example, in Perl:
#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
my $nk = 1152;
my $nb = 24;
for my $k (0 .. $nk - 1) {
for my $i (0 .. $nb - 1) {
for my $j (0 .. $nb - 1) {
say "$k\t$i\t$j"
}
}
}
The original bash solution runs for 22 seconds, the Perl one finishes in 0.1 seconds. The output is identical.
#Jacek : I don't think the I/O is the problem, but the number of child processes spawned. I would store the result of the seq 0 $((nb-1)) into an array and loop over the array, i.e.
nb_seq=( $(seq 0 $((nb-1)) )
...
for i in "${nb_seq[#]}"; do
for j in "${nb_seq[#]}"; do
seq is bad) once i've done this function special for this case:
$ que () { printf -v _N %$1s; _N=(${_N// / 1}); printf "${!_N[*]}"; }
$ que 10
0 1 2 3 4 5 6 7 8 9
And you can try to write first all to a var and then whole var into a file:
store+="$k\t$i\t$j\n"
printf "$store" > file
No. it's even worse like that)
I am trying to read a file in bash forloop. But I do not know how to put write the script for this.
for i in $( seq 0 $step 10 )
do
echo "Rendering: "$(( i + j ))
python auto_fine.py density000000.vtu velocity000000.vtu $(( i + j ))
done
each and every loop I need to call
i -> 0 python auto_fine.py density000000.vtu velocity000000.vtu
i -> 1 python auto_fine.py density000010.vtu velocity000010.vtu
i -> 2 python auto_fine.py density000020.vtu velocity000020.vtu
It seems to me that you need to zero pad the numbers sed provides to you:
As seen in How to zero pad a sequence of integers in bash so that all have the same width?, you need to do something like
$ seq -f "%06g" 0 10 100
Which returns:
000000
000010
000020
...
000100
All together,
for i in $(seq -f "%06g" 0 10 100)
do
# echo "Rendering: "$(( i + j )) -- not sure what this does
python auto_fine.py density$i.vtu velocity$i.vtu
done
Bash can do this without requiring external tools like seq.
for i in {0..100}; do
[[ $i = *0 ]] || continue
python auto_fine.py density$(printf '%06d' $i).vtu velocity$(printf '%06d' $i).vtu
done
This uses pattern matching (*0) to limit your list to every 10 numbers, which is a bit of a hack, but will work against your sample data.
You could alternately loop against your zero-padded numeric strings directly:
for i in $(printf '%05d0 ' {0..10}); do
python auto_fine.py density$i.vtu velocity$i.vtu
done
This option shows you every 10 items by placing a zero in the printf format after the incrementing number, which becomes the tens digit. If you want more arbitrary sequencing, you might use multipliers, still without spawning external processes:
low=0
high=100
mult=10
for i in $(eval echo {$low..$((high/mult))}); do
n=$(printf '%06d' $((i*mult)))
python auto_fine.py density$n.vtu velocity$n.vtu
done
Note the eval, which lets you expand variables for use in your sequence expression. (If you are getting these numbers from an external source, have your script validate them before using them!)
If you're using bash version 4 (i.e. not the native version on OSX), you also have increments available in sequence expressions. From the man page:
A sequence expression takes the form {x..y[..incr]}, where x and y are
either integers or single characters, and incr, an optional increment,
is an integer.
So perhaps:
low=0
high=100
mult=10
for i in $(eval "printf '%06d ' {$low..$high..$mult}"); do
python auto_fine.py density$i.vtu velocity$i.vtu
done
Note that in sequence expressions, the first member of the sequence is the first number provided, rather than merely a product of a multiplier. We have quotes around the printf to ensure that the sequence expression is expanded by eval, and not interpreted by the command substitution ($(..)).
looping for all the files in the current dir is trivial:
for i in $( ls -1 )
do
# your code here, variable is referenced with $i
done
what's the j variable you are using?
What's an easy way to read random line from a file in a shell script?
You can use shuf:
shuf -n 1 $FILE
There is also a utility called rl. In Debian it's in the randomize-lines package that does exactly what you want, though not available in all distros. On its home page it actually recommends the use of shuf instead (which didn't exist when it was created, I believe). shuf is part of the GNU coreutils, rl is not.
rl -c 1 $FILE
Another alternative:
head -$((${RANDOM} % `wc -l < file` + 1)) file | tail -1
sort --random-sort $FILE | head -n 1
(I like the shuf approach above even better though - I didn't even know that existed and I would have never found that tool on my own)
This is simple.
cat file.txt | shuf -n 1
Granted this is just a tad slower than the "shuf -n 1 file.txt" on its own.
perlfaq5: How do I select a random line from a file? Here's a reservoir-sampling algorithm from the Camel Book:
perl -e 'srand; rand($.) < 1 && ($line = $_) while <>; print $line;' file
This has a significant advantage in space over reading the whole file in. You can find a proof of this method in The Art of Computer Programming, Volume 2, Section 3.4.2, by Donald E. Knuth.
using a bash script:
#!/bin/bash
# replace with file to read
FILE=tmp.txt
# count number of lines
NUM=$(wc - l < ${FILE})
# generate random number in range 0-NUM
let X=${RANDOM} % ${NUM} + 1
# extract X-th line
sed -n ${X}p ${FILE}
Single bash line:
sed -n $((1+$RANDOM%`wc -l test.txt | cut -f 1 -d ' '`))p test.txt
Slight problem: duplicate filename.
Here's a simple Python script that will do the job:
import random, sys
lines = open(sys.argv[1]).readlines()
print(lines[random.randrange(len(lines))])
Usage:
python randline.py file_to_get_random_line_from
Another way using 'awk'
awk NR==$((${RANDOM} % `wc -l < file.name` + 1)) file.name
A solution that also works on MacOSX, and should also works on Linux(?):
N=5
awk 'NR==FNR {lineN[$1]; next}(FNR in lineN)' <(jot -r $N 1 $(wc -l < $file)) $file
Where:
N is the number of random lines you want
NR==FNR {lineN[$1]; next}(FNR in lineN) file1 file2
--> save line numbers written in file1 and then print corresponding line in file2
jot -r $N 1 $(wc -l < $file) --> draw N numbers randomly (-r) in range (1, number_of_line_in_file) with jot. The process substitution <() will make it look like a file for the interpreter, so file1 in previous example.
#!/bin/bash
IFS=$'\n' wordsArray=($(<$1))
numWords=${#wordsArray[#]}
sizeOfNumWords=${#numWords}
while [ True ]
do
for ((i=0; i<$sizeOfNumWords; i++))
do
let ranNumArray[$i]=$(( ( $RANDOM % 10 ) + 1 ))-1
ranNumStr="$ranNumStr${ranNumArray[$i]}"
done
if [ $ranNumStr -le $numWords ]
then
break
fi
ranNumStr=""
done
noLeadZeroStr=$((10#$ranNumStr))
echo ${wordsArray[$noLeadZeroStr]}
Here is what I discovery since my Mac OS doesn't use all the easy answers. I used the jot command to generate a number since the $RANDOM variable solutions seems not to be very random in my test. When testing my solution I had a wide variance in the solutions provided in the output.
RANDOM1=`jot -r 1 1 235886`
#range of jot ( 1 235886 ) found from earlier wc -w /usr/share/dict/web2
echo $RANDOM1
head -n $RANDOM1 /usr/share/dict/web2 | tail -n 1
The echo of the variable is to get a visual of the generated random number.
Using only vanilla sed and awk, and without using $RANDOM, a simple, space-efficient and reasonably fast "one-liner" for selecting a single line pseudo-randomly from a file named FILENAME is as follows:
sed -n $(awk 'END {srand(); r=rand()*NR; if (r<NR) {sub(/\..*/,"",r); r++;}; print r}' FILENAME)p FILENAME
(This works even if FILENAME is empty, in which case no line is emitted.)
One possible advantage of this approach is that it only calls rand() once.
As pointed out by #AdamKatz in the comments, another possibility would be to call rand() for each line:
awk 'rand() * NR < 1 { line = $0 } END { print line }' FILENAME
(A simple proof of correctness can be given based on induction.)
Caveat about rand()
"In most awk implementations, including gawk, rand() starts generating numbers from the same starting number, or seed, each time you run awk."
-- https://www.gnu.org/software/gawk/manual/html_node/Numeric-Functions.html