I'm trying to run a command using different variables as parameters. This is written as a bash script.
for i, j, k in $(seq 2 0.1 6), $(seq 2 0.25 5.5), $(seq 1 1 10)
do
p.p_s_e r=100 a_t=S res=$i lam=$j s=$k sig=10 >> $k_lam_$j_res_$i.log
p.p_s_e r=100 a_t=S res=$i lam=$j s=$k sig=20 >> $k_lam_$j_res_$i.log
p.p_s_e r=100 a_t=S res=$i lam=$j s=$k sig=40 >> $k_lam_$j_res_$i.log
done
When I run this, the program does not take any of the values I am trying to give it. Sorry I can't be more clear about what I am trying to do. p.p_s_e is the program, the following X=y are variables, and I need to output to be written into a file. I think it's the way I am using for, do, done loops.
You just need three loops. (Four, if you add a loop to iterate over the sig values as well.)
for i in $(seq 2 0.1 6); do
for j in $(seq 2 0.25 5.5); do
for k in $(seq 1 1 10); do
for sig in 10 20 40; do
p.p_s_e r=100 a_t=S res=$1 lam=$j s=$k sig=$sig
done >> ${k}_lam_${j}_res_${i}.log
done
done
done
Related
I have a for loop in bash that writes values to a file. However, because there are a lot of values, the process takes a long time, which I think can be saved by improving the code.
nk=1152
nb=24
for k in $(seq 0 $((nk-1))); do
for i in $(seq 0 $((nb-1))); do
for j in $(seq 0 $((nb-1))); do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
I've moved the output action to after the entire loop is done rather than echo -e "$k\t$i\t$j" >> file.dat to avoid opening and closing the file many times. However, the speed the script writes to the file is still rather slow, ~ 10kbps.
Is there a better way to improve the IO?
Many thanks
Jacek
It looks like the seq calls are fairly punishing since that is a separate process. Try this just using shell math instead:
for ((k=0;k<=$nk-1;k++)); do
for ((i=0;i<=$nb-1;i++)); do
for ((j=0;j<=$nb-1;j++)); do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
It takes just 7.5s on my machine.
Another way is to compute the sequences just once and use them repeatedly, saving a lot of shell calls:
nk=1152
nb=24
kseq=$(seq 0 $((nk-1)))
bseq=$(seq 0 $((nb-1)))
for k in $kseq; do
for i in $bseq; do
for j in $bseq; do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
This is not really "better" than the first option, but it shows how much of the time is spent spinning up instances of seq versus actually getting stuff done.
Bash isn't always the best for this. Consider this Ruby equivalent which runs in 0.5s:
#!/usr/bin/env ruby
nk=1152
nb=24
nk.times do |k|
nb.times do |i|
nb.times do |j|
puts "%d\t%d\t%d" % [ k, i, j ]
end
end
end
What is the most time consuming is calling seq in a nested loop. Keep in mind that each time you call seq it loads command from disk, fork a process to run it, capture the output, and store the whole output sequence into memory.
Instead of calling seq you could use an arithmetic loop:
#!/usr/bin/env bash
declare -i nk=1152
declare -i nb=24
declare -i i j k
for ((k=0; k<nk; k++)); do
for (( i=0; i<nb; i++)); do
for (( j=0; j<nb; j++)); do
printf '%d\t%d\t%d\n' "$k" "$i" "$j"
done
done
done > file.dat
Running seq in a subshell consumes most of the time.
Switch to a different language that provides all the needed features without shelling out. For example, in Perl:
#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
my $nk = 1152;
my $nb = 24;
for my $k (0 .. $nk - 1) {
for my $i (0 .. $nb - 1) {
for my $j (0 .. $nb - 1) {
say "$k\t$i\t$j"
}
}
}
The original bash solution runs for 22 seconds, the Perl one finishes in 0.1 seconds. The output is identical.
#Jacek : I don't think the I/O is the problem, but the number of child processes spawned. I would store the result of the seq 0 $((nb-1)) into an array and loop over the array, i.e.
nb_seq=( $(seq 0 $((nb-1)) )
...
for i in "${nb_seq[#]}"; do
for j in "${nb_seq[#]}"; do
seq is bad) once i've done this function special for this case:
$ que () { printf -v _N %$1s; _N=(${_N// / 1}); printf "${!_N[*]}"; }
$ que 10
0 1 2 3 4 5 6 7 8 9
And you can try to write first all to a var and then whole var into a file:
store+="$k\t$i\t$j\n"
printf "$store" > file
No. it's even worse like that)
From 3 sequences (for example):
seq 0 0.2 1,
seq 0 0.3 1.5,
seq 0 0.5 1
I want to generate something like
0:0:0 0:0:0.5 0:0:1 0:0.3:0 0:0.3:0.5 0:0.3:1.... They are in a format a:b:c where a is from the first sequence, b from the second, c from the third and all the combination show up once.
If these are integers and with unity step, I could use {1..10}:{2..10}:{3..10}, and it works nicely, but is there anyway to extend this brace function to noninterger and non unity step?
Many thanks!
Honestly, awk is a better tool than seq for this job; it's POSIX-standardized, much faster at I/O than bash is, and you can run only one instance to generate all your output:
awk '
BEGIN {
for(a=0; a<=1; a+=0.2) {
for(b=0; b<=1.5; b+=0.2) {
for(c=0; c<=1; c+=0.5) {
printf("%.1f:%.1f:%.1f\n", a, b, c);
}
}
}
}' </dev/null
However, if for some reason you really want to use seq, nesting three BashFAQ #1 while read loops will do the job:
#!/usr/bin/env bash
while read -r a; do
while read -r b; do
while read -r c; do
printf '%.1f:%.1f:%.1f\n' "$a" "$b" "$c"
done < <(seq 0 0.5 1)
done < <(seq 0 0.3 1.5)
done < <(seq 0 0.2 1)
On my system, the seq version runs in ~0.3 seconds wall-clock, whereas the awk version takes ~0.01s.
This might work for you (GNU sed & parallel):
parallel echo ::: $(seq 0 0.2 1) ::: $(seq 0 0.3 1.5) ::: $(seq 0 0.5 1) |
sed -z 'y/ \n/: /;s/\.0//g'
An alternative,using bash and sed:
echo {00..10..2}:{00..15..3}:{00..10..5} | sed 's/\B/./g;s/\.0//g'
Some of you are probably familiar with Project Euler, and I'm currently attempting a few of their problems to teach myself some more bash. They're a bit more mathematical than 'script-y' but it helps with syntax etc.
The problem currently asks me to solve:
If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.
Find the sum of all the multiples of 3 or 5 below 1000.
The code I have looks like so:
#!/bin/bash
i="1"
for i in `seq 1 333`
do
threes[$i]=`calc $i*3` # where 'calc' is a function written in bashrc
#calc actually looks like: calc() {awk "BEGIN { print "$*"} }
let "sumthrees = sumthrees + ${threes[$i]}"
done
for i in `seq 1 199`
do
fives[$i]=`calc $i*5`
let "sumfives = sumfives + ${fives[$i]}"
done
let "ans = $sumfives + $sumthrees"
echo "The sum of all 3 factors is $sumthrees and the sum of all five factors is $sumfives"
echo "The sum of both is $ans"
#So I can repeatedly run the script without bash remembering the variables between executions
unset i
unset fives
unset threes
unset sumfives
unset sumthrees
unset ans
So far I've not gotten the correct answer, but have run out of ideas as to where I'm going wrong. (FYI, the script currently gives me 266333, which I believe is close, but I don't know the answer yet.)
Can anyone spot anything? And for my own learning, if there are more elegant solutions to this that people might like to share that would be great.
EDIT
Thanks for all the answers, super informative. Since there are so many useful answers here I'll accept my favourite as the proper thread answer.
Blue Moon pointed out the actual problem with your logic.
You don't need to store all the threes and fives in arrays because you don't need them later.
You don't need to unset variables at the end of a script if you use ./yourscript or bash script because they'll disappear along with the
shell instance (better to initialize them first in any case).
You don't need awk to do math, bash does that just fine.
seq and let are not the best way to do anything in a bash script.
Here's a straight forward version:
#!/bin/bash
sum=0
for ((i=1; i<1000; i++))
do
if (( i%3 == 0 || i%5 == 0 ))
then
(( sum += i ))
fi
done
echo "$sum"
Your logic is almost right except that there are numbers which divide by both 3 and 5. So you are adding these numbers twice. Hence, you get wrong answer.
Use another loop similar to ones you have and subtract the ones that divide by both 3 and 5 from the result.
A few tips you might find useful:
In bash, you use let to give the shell a hint that a variable should be considered a number. All bash variables are strings, but you can do arithmetic on numerical strings. If I say let i=1 then i is set to 1, but if I say let i="taco" then $i will be 0, because it couldn't be read as a number. You can achieve a small amount of type-safety when doing mathematical work in the shell.
Bash also has $((this)) mechanism for doing math! You can check it out yourself: echo $((2 + 2)) -> 4, and even more relevant to this problem: echo $((6 % 3 == 0)) -> 1
In case you aren't familiar, % divides the first number by the second, and gives back the remainder; when the remainder is 0, it means that the first is divisible by the second! == is a test to see if two things are equal, and for logical tests like this 1 represents true and 0 represents false. So I'm testing if 6 is divisible by 3, which it is, and the value I get back is 1.
The test brackets, [ ... ] have a "test for equality" flag, -eq, which you can use to check if a math expression has a certain value (man test for more details)!
$ let i=6
$ echo $((i % 3 == 0 || i % 5 == 0))
1
$ if [ $((i % 3 == 0 || i % 5 == 0)) -eq 1 ]; then echo "yes"; fi
yes
(|| is another logical test - $((a || b)) will be 1 (true) when a is true or b is true).
Finally, instead of doing this for the number 6, you could do it in a for loop and increment a sum variable every time you find a multiple of 3 or 5:
let sum=0
for i in {1..1000}; do
if [ $((i % 3 == 0 || i % 5 == 0)) -eq 1 ]; then
let sum=$((sum + i))
fi
done
echo $sum
And there you'd have a working solution!
Bash has a lot of nice little tricks, (and a lot more mean ugly tricks), but it's worth learning at least a handful of them to make use of it as a scripting tool.
How about creative use of the modulus function & some checks. Then you have just 1 loop.
#!/bin/bash
i=1
while [ $i -lt 1000 ]
do
if [ $(($i % 3)) -eq 0 ] || [ $(($i % 5)) -eq 0 ]
then
sumall=$(($sumall+$i))
fi
i=$(($i+1))
done
echo "The sum of both is $sumall"
Answer: 233168
A different solution:
#!/bin/bash
sum=0
for n in {1..999}; do [ $(((n%5) * (n%3))) -eq 0 ] && sum=$((sum+n)); done
echo $sum
The script loops through all numbers below 1000, tests if the product of the number mod 3 and the number mod 5 is 0 (the product of two numbers can only be zero if one of them is zero). If that is the case, it adds the current number to a sum, which is printed out afterwards.
By the way, if I were you I'd include the definition of the calc function inside the script, to get a self-contained solution that doesn't need your specific configuration.
I need to add a non-integer counter value of a loop to the counter of another loop. like what follows:
I have a two loops like:
numi=$(awk 'BEGIN{for(i=0;i<=.4;i+=0.2)print i}')
numj=$(awk 'BEGIN{for(j=.1;j<=1;i+=0.3)print j}')
for i in $numi
do
for j in $numj
do
***then here I want to change j eg. to j+i and do blah blah but
the problem is that bash does not like float and I can't find a way
to pass the float value. As I'm running another software through this
loop and use the value to feed that software I have to do it in bash
or somehow pass the values to bash.
j+i and do blah blah but the problem is that bash does not like float
if this is your concern, see if below example helps:
$ bc<<<'3.5+1'
4.5
for i in $(seq 0 .2 .4)
do
for j in $(seq .1 .3 1)
do
j_and_i=$(($i+$j))
echo $j_and_i
done
done
The $() construction executes the command within parentheses and substitutes the results into your line. The seq command prints a sequence of numbers. In these cases it's given a FIRST, INCREMENT, and LAST values to use. seq .1 .3 1 results in "0.1 0.4 0.7 1.0".
The $(()) construction does math. $((1+1)) is 2, for example.
I'd like to apologize if my question had already been asked, but english isn't my native language and I didn't find the answer. I'd like to have a bash script that executes a program I'll call MyProgram, and I want it to run with a fixed number of arguments which consist in random numbers. I'd like to have something like this:
./MyProgram for(i = 0; i < 1000; i++) $(($RANDOM%200-100))
How should I go about this?
You (mostly) just have the loop and the actual program call inverted.
for ((i=0; i < 1000; i++)); do
./MyProgram $((RANDOM%200 - 100))
done
If, however, you actually want 1000 different arguments passed to a single call, you have to build up a list first.
args=()
for ((i=0; i < 1000; i++)); do
args+=( $((RANDOM%200 - 100)) )
done
./MyProgram "${args[#]}"
The
$RANDOM % 200 - 100
is the same as the next perl
perl -E 'say int(200*rand() -100) for (1..1000)'
e.g. the
perl -E 'say int(200*rand() -100) for (1..1000)' | xargs -n1 ./MyProgram
will run like:
./MyProgram -10
./MyProgram 13
... 1000 times ...
./MyProgram 55
./MyProgram -31
if you need 1000 args
./MyProgram $(perl -E 'say int(200*rand() -100) for (1..1000)')
will produce
./MyProgram 5 -41 -81 -79 -14 ... 1000 numbers ... -63 -9 95 -9 -29
In addition to what #chepner says, you can also use the for ... in style of for loop. This looks like:
for a in one two three; do
echo "${a}"
done
which would produce the result:
one
two
three
In other words, the list of words after the in part, separated by spaces, is looped over, with each iteration of the loop having a different word in the variable a.
To call your program 1000 times (or just modify to produce the list of arguments to run it once as in #chepner's answer) you could then do:
for a in $(seq 1 1000); do
./MyProgram $((RANDOM%200 - 100))
done
where the output of the seq command is providing the list of values to loop over. Although the traditional for loop may be more immediately obvious to many programmers, I like for ... in because it can be applied in lots of situations. A crude and mostly pointless ls, for example:
for a in *; do
echo "${a}"
done
for ... in is probably the bit of "advanced" bash that I find the most useful, and make use of it very frequently.