bash script loop multiple variables - bash

I am trying to write something like following
for i in {a..z} && j in {1..26}
do
echo "/dev/sd"$i"1 /disk$j ext4 noatime 1 1" >> test
done
Of course this is not correct syntax. Can someone please help me with correct syntax for doing this?

To be generic, you can use 'length' as shown below.
#!/bin/bash
# Define the arrays
array1=("a" "b" "c" "d")
array2=("w" "x" "y" "z")
# get the length of the arrays
length=${#array1[#]}
# do the loop
for ((i=0;i<=$length;i++)); do
echo -e "${array1[$i]} : ${array2[$i]}"
done
You can also assign the array like the following
array1=`awk -F" " '$1 == "CLIENT" { print $2 }' clientserver.lst`

You can use arrays for that:
A=({a..z}) B=({1..26})
for (( I = 0; I < 26; ++I )); do
echo "/dev/sd${A[I]} /disk${B[I]} ext4 noatime 1 1" >> test
done
Example output:
/dev/sda /disk1 ext4 noatime 1 1
...
/dev/sdz /disk26 ext4 noatime 1 1
Update:
As suggested you could just use the index for values of B:
A=('' {a..z})
for (( I = 1; I <= 26; ++I )); do
echo "/dev/sd${A[I]} /disk${I} ext4 noatime 1 1" >> test
done
Also you could do some formatting with printf to get a better output, and cleaner code:
A=('' {a..z})
for (( I = 1; I <= 26; ++I )); do
printf "%s%20s%15s%15s%4s%2s\n" "/dev/sd${A[I]}" "/disk${I}" ext4 noatime 1 1 >> test
done
Also, if you don't intend to append data to file, but only write once every generated set of lines, just make redirection by block instead:
A=('' {a..z})
for (( I = 1; I <= 26; ++I )); do
printf "%s%20s%15s%15s%4s%2s\n" "/dev/sd${A[I]}" "/disk${I}" ext4 noatime 1 1
done > test

Related

How do I create large CSVs in seconds?

I am trying to create 1000s of large CSVs rapidly. This function generates the CSVs:
function csvGenerator () {
for ((i=1; i<=$NUMCSVS; i++)); do
CSVNAME=$DIRNAME"-"$CSVPREFIX$i$CSVEXT
HEADERARRAY=()
if [[ ! -e $CSVNAME ]]; then #Only create csv file if it not exist
touch $CSVNAME
echo "file: "$CSVNAME "created at $(date)" >> ../status.txt
fi
for ((j=1; j<=$NUMCOLS; j++)); do
if (( j < $NUMCOLS )) ; then
HEADERNAME=$DIRNAME"-csv-"$i"-header-"$j", "
elif (( j == $NUMCOLS )) ; then
HEADERNAME=$DIRNAME"-csv-"$i"-header-"$j
fi
HEADERARRAY+=$HEADERNAME
done
echo $HEADERARRAY > $CSVNAME
for ((k=1; k<=$NUMROWS; k++)); do
ROWARRAY=()
for ((l=1; l<=$NUMCOLS; l++)); do
if (( l < $NUMCOLS )) ; then
ROWVALUE=$DIRNAME"-csv-"$i"-r"$k"c"$l", "
elif (( l == $NUMCOLS )) ; then
ROWVALUE=$DIRNAME"-csv-"$i"-r"$k"c"$l
fi
ROWARRAY+=$ROWVALUE
done
echo $ROWARRAY >> $CSVNAME
done
done
}
The script takes ~3 mins to generate a CSV with 100k rows and 70 cols. What do I need to do to generate these CSVs at the rate of 1 CSV/~10 seconds?
Let me start by saying that bash and "performant" don't usually go together in the same sentence. As other commentators suggested, awk may be a good choice that's adjacent in some senses.
I haven't yet had a chance to run your code, but it opens and closes the output file once per row — in this example, 100,000 times. Each time it must seek to the end of the file so that it can append the latest row.
Try pulling the actual generation (everything after for ((j=1; j<=$NUMCOLS; j++)); do) into a new function, like generateCsvContents. In that new function, don't reference $CSVNAME, and remove the redirections on the echo statements. Then, in the original function, call the new function and redirect its output to the filename. Roughly:
function csvGenerator () {
for ((i=1; i<=NUMCSVS; i++)); do
CSVNAME=$DIRNAME"-"$CSVPREFIX$i$CSVEXT
if [[ ! -e $CSVNAME ]]; then #Only create csv file if it not exist
echo "file: $CSVNAME created at $(date)" >> ../status.txt
fi
# This will create $CSVNAME if it doesn't yet exist
generateCsvContents > "$CSVNAME"
done
}
function generateCsvContents() {
HEADERARRAY=()
for ((j=1; j<=NUMCOLS; j++)); do
if (( j < NUMCOLS )) ; then
HEADERNAME=$DIRNAME"-csv-"$i"-header-"$j", "
elif (( j == NUMCOLS )) ; then
HEADERNAME=$DIRNAME"-csv-"$i"-header-"$j
fi
HEADERARRAY+=$HEADERNAME
done
echo $HEADERARRAY
for ((k=1; k<=NUMROWS; k++)); do
ROWARRAY=()
for ((l=1; l<=NUMCOLS; l++)); do
if (( l < NUMCOLS )) ; then
ROWVALUE=$DIRNAME"-csv-"$i"-r"$k"c"$l", "
elif (( l == NUMCOLS )) ; then
ROWVALUE=$DIRNAME"-csv-"$i"-r"$k"c"$l
fi
ROWARRAY+=$ROWVALUE
done
echo "$ROWARRAY"
done
}
"Not this way" is I think the answer.
There are a few problems here.
You're not using your arrays as arrays. When you treat them like strings, you affect only the first element in the array, which is misleading.
The way you're using >> causes the output file to be opened and closed once for every line. That's potentially wasteful.
You're not quoting your variables. In fact, you're quoting the stuff that doesn't need quoting, and not quoting the stuff that does.
Upper case variable names are not recommended, due to the risk of collision with system variables. ref
Bash isn't good at this. Really.
A cleaned up version of your function might look like this:
csvGenerator2() {
for (( i=1; i<=NUMCSVS; i++ )); do
CSVNAME="$DIRNAME-$CSVPREFIX$i$CSVEXT"
# Only create csv file if it not exist
[[ -e "$CSVNAME" ]] && continue
touch "$CSVNAME"
date "+[%F %T] created: $CSVNAME" | tee -a status.txt >&2
HEADER=""
for (( j=1; j<=NUMCOLS; j++ )); do
printf -v HEADER '%s, %s-csv-%s-header-%s' "$HEADER" "$DIRNAME" "$i" "$j"
done
echo "${HEADER#, }" > "$CSVNAME"
for (( k=1; k<=NUMROWS; k++ )); do
ROW=""
for (( l=1; l<=NUMCOLS; l++ )); do
printf -v ROW '%s, %s-csv-%s-r%sc%s' "$ROW" "$DIRNAME" "$i" "$k" "$l"
done
echo "${ROW#, }"
done >> "$CSVNAME"
done
}
(Note that I haven't switched the variables to lower case because I'm lazy, but it's still a good idea.)
And if you were to make something functionally equivalent in awk:
csvGenerator3() {
awk -v NUMCSVS="$NUMCSVS" -v NUMCOLS="$NUMCOLS" -v NUMROWS="$NUMROWS" -v DIRNAME="$DIRNAME" -v CSVPREFIX="$CSVPREFIX" -v CSVEXT="$CSVEXT" '
BEGIN {
for ( i=1; i<=NUMCSVS; i++) {
out=sprintf("%s-%s%s%s", DIRNAME, CSVPREFIX, i, CSVEXT)
if (!system("test -e " CSVNAME)) continue
system("date '\''+[%F %T] created: " out "'\'' | tee -a status.txt >&2")
comma=""
for ( j=1; j<=NUMCOLS; j++ ) {
printf "%s%s-csv-%s-header-%s", comma, DIRNAME, i, j > out
comma=", "
}
printf "\n" >> out
for ( k=1; k<=NUMROWS; k++ ) {
comma=""
for ( l=1; l<=NUMCOLS; l++ ) {
printf "%s%s-csv-%s-r%sc%s", comma, DIRNAME, i, k, l >> out
comma=", "
}
printf "\n" >> out
}
}
}
'
}
Note that awk does not suffer from the same open/closer overhead mentioned earlier with bash; when a file is used for output or as a pipe, it gets opened once and is left open until it is closed.
Comparing the two really highlights the choice you need to make:
$ time bash -c '. file; NUMCSVS=1 NUMCOLS=10 NUMROWS=100000 DIRNAME=2 CSVPREFIX=x CSVEXT=.csv csvGenerator2'
[2019-03-29 23:57:26] created: 2-x1.csv
real 0m30.260s
user 0m28.012s
sys 0m1.395s
$ time bash -c '. file; NUMCSVS=1 NUMCOLS=10 NUMROWS=100000 DIRNAME=3 CSVPREFIX=x CSVEXT=.csv csvGenerator3'
[2019-03-29 23:58:23] created: 3-x1.csv
real 0m4.994s
user 0m3.297s
sys 0m1.639s
Note that even my optimized bash version is only a little faster than your original code.
Refactoring your two inner for-loops to loops like this will save time:
for ((j=1; j<$NUMCOLS; ++j)); do
HEADERARRAY+=$DIRNAME"-csv-"$i"-header-"$j", "
done
HEADERARRAY+=$DIRNAME"-csv-"$i"-header-"$NUMCOLS

Two-level 'for' loop of a customised list

Using a two-level for loop and seq works fine, and the code
for i in `seq 0 3`; do for j in `seq 0 3`; do echo $i $j; done; done
gives the expected output:
0 0
0 1
1 0
1 1
But if I want a more customised list of numbers:
for i in '-1 4.5'; do for j in '0 -2.2'; do echo $i $j; done; done
I get the output
-1 4.5 0 -2.2
Is there an easy way to do this?
The usage of single ticks is preventing the shell from tokenizing the list supplied to the for loop:
#!/bin/sh
for i in -1 4.5
do for j in 0 -2.2
do # -- keeps the highly portable printf utility from
# interpreting '-' as an argument
printf -- "$i" "$j\n"
done
done
var a = [1,2,3];
var b = [1,2,3];
for (i in a) {
for (j in b) {
alert (i + j);
}
}
I have also created a jsfiddle:
http://jsfiddle.net/f8k5tpqm/1/

Bash Script accepting a number, then printing a set of int from 0 to the number entered

I am trying to write a bash script that accepts a number from the keyboard, and then prints a set of integers from 0 to the number entered. I can't figure out how to do it at all.
This is my code:
while [ 1 ]
do
echo -n "Enter a color: "
read user_answer
for (( $user_answer = $user_answer; $user_answer>0; $user_answer--))
echo $user_answer
fi
done
exit
The error I'm recieving is:
number_loop: line 10: syntax error near unexpected token echo'
number_loop: line 10: echo $user_answer'
Assign a separate variable in order to use increment/decrement operators. $user_answer=$user_answer will always be true and it will throw an error when trying to use decrement. Try the following :
#!/bin/bash
while [ 1 ]
do
echo -n "Enter a color: "
read user_answer
for (( i=$user_answer; i>0; i-- ))
do
echo $i
done
done
exit
You missed the do statement between your for and the echo.
bash has many options to write numbers. What you seem to be trying to do is easiest done with seq:
seq $user_answer -1 0
If you want to use your loop, you have to insert a ; do and replace the fi with done, and replace several $user_answer:
for (( i = $user_answer; i>0; i--)); do
echo $i
done
(btw: I assumed that you wanted to write the numbers in reverse order, as you are going backwards in your loop. Forwards is even easier with seq:
seq 0 $user_input
)
This is where a c-style loop works particularly well:
#!/bin/bash
for ((i = 1; i <= $1; i++)); do
printf "%s\n" "$i"
done
exit 0
Example
$ bash simplefor.sh 10
1
2
3
4
5
6
7
8
9
10
Note: <= is used as the for loop test so it 10 it iterates 1-10 instead of 0-9.
In your particular case, iterating from $user_answer you would want:
for (( i = $user_answer; i > 0; i--)); do
echo $i
done
The for loop is a bash internal command, so it doesn't fork a new process.
The seq command has a nice, one-line syntax.
To get the best of the twos, you can use the { .. } syntax:
eval echo {1..$answer}

slow running script. How can I increase its speed?

How can I speed this up? it's taking about 5 minutes to make one file...
it runs correctly, but I have a little more than 100000 files to make.
Is my implementation of awk or sed slowing it down? I could break it down into several smaller loops and run it on multiple processors but one script is much easier.
#!/bin/zsh
#1000 configs per file
alpha=( a b c d e f g h i j k l m n o p q r s t u v w x y z )
m=1000 # number of configs per file
t=1 #file number
for (( i=1; i<=4; i++ )); do
for (( j=i; j<=26; j++ )); do
input="arc"${alpha[$i]}${alpha[$j]}
n=1 #line number
#length=`sed -n ${n}p $input| awk '{printf("%d",$1)}'`
#(( length= $length + 1 ))
length=644
for ((k=1; k<=$m; k++ )); do
echo "$hmbi" >> ~/Glycine_Tinker/configs/config$t.in
echo "jobtype = energy" >> ~/Glycine_Tinker/configs/config$t.in
echo "analyze_only = false" >> ~/Glycine_Tinker/configs/config$t.in
echo "qm_path = qm_$t" >> ~/Glycine_Tinker/configs/config$t.in
echo "mm_path = aiff_$t" >> ~/Glycine_Tinker/configs/config$t.in
cat head.in >> ~/Glycine_Tinker/configs/config$t.in
water=4
echo $k
for (( l=1; l<=$length; l++ )); do
natom=`sed -n ${n}p $input| awk '{printf("%d",$1)}'`
number=`sed -n ${n}p $input| awk '{printf("%d",$6)}'`
if [[ $natom -gt 10 && $number -gt 0 ]]; then
symbol=`sed -n ${n}p $input| awk '{printf("%s",$2)}'`
x=`sed -n ${n}p $input| awk '{printf("%.10f",$3)}'`
y=`sed -n ${n}p $input| awk '{printf("%.10f",$4)}'`
z=`sed -n ${n}p $input| awk '{printf("%.10f",$5)}'`
if [[ $water -eq 4 ]]; then
echo "--" >> ~/Glycine_Tinker/configs/config$t.in
echo "0 1 0.4638" >> ~/Glycine_Tinker/configs/config$t.in
water=1
fi
echo "$symbol $x $y $z" >> ~/Glycine_Tinker/configs/config$t.in
(( water= $water + 1 ))
fi
(( n= $n + 1 ))
done
cat tail.in >> ~/Glycine_Tinker/configs/config$t.in
(( t= $t + 1 ))
done
done
done
One thing that is going to be killing you here is the sheer number of processes being created. Especially when they are doing the exact same thing.
Consider doing the sed -n ${n}p $input once per loop iteration.
Also consider doing the equivalent of awk as a shell array assignment, then accessing the individual elements.
With these two things you should be able to get the 12 or so processes (and the shell invocation via back quotes) down to a single shell invocation and the backquote.
Obviously, Ed's advice is far preferable, but if you don't want to follow that, I had a couple of thoughts...
Thought 1
Rather than run echo 5 times and cat head.in onto the Glycine file, each of which causes the file to be opened, seeked (or sought maybe) to the end, and appended, you could do that in one go like this:
# Instead of
hmbi=3
echo "$hmbi" >> ~/Glycine_thing
echo "jobtype = energy" >> ~/Glycine_thing
echo "somethingelse" >> ~/Glycine_thing
echo ... >> ~/Glycine_thing
echo ... >> ~/Glycine_thing
cat ... >> ~/Glycine_thing
# Try this
{
echo "$hmbi"
echo "jobtype = energy"
echo "somethingelse"
echo
echo
cat head.in
} >> ~/Glycine_thing
# Or, better still, this
echo -e "$hmbi\njobtype = energy\nsomethingelse" >> Glycine_thing
# Or, use a here-document, as suggested by #mklement0
cat -<<EOF >>Glycine
$hmbi
jobtype = energy
next thing
EOF
Thought 2
Rather than invoke sed and awk 5 times to find 5 parameters, just let awk do what sed was doing, and also do all 5 things in one go:
read symbol x y z < <(awk '...{printf "%.10f %.10f %.10f" $2,$3,$4}' $input)

Comparison between array items

I've written a script to calculate the bandwidth usage of an OpenVZ container over time and suspend it if it uses too much too quickly. Here is the script so far:
#!/bin/bash
# Thresholds are in bytes per second
LOGDIR="/var/log/outbound_ddos"
THRESHOLD1=65536
THRESHOLD2=117964
while [ 1 ]
do
for veid in $(/usr/sbin/vzlist -o veid -H)
do
# Create the log file if it doesn't already exist
if ! test -e $LOGDIR/$veid.log; then
touch $LOGDIR/$veid.log
fi
# Parse out the inbound/outbound traffic and assign them to the corresponding variables
eval $(/usr/sbin/vzctl exec $veid "grep venet0 /proc/net/dev" | \
awk -F: '{print $2}' | awk '{printf"CTOUT=%s\n", $9}')
# Print the output and a timestamp to a log file
echo $(date +%s) $CTOUT >> $LOGDIR/$veid.log
# Read last 10 entries into arrays
i=0
tail $LOGDIR/$veid.log | while read time byte
do
times[i]=$time
bytes[i]=$byte
let ++i
done
# Time checks & calculations for higher threshold
counter=0
for (( i=0; i<9; i++ ))
do
# If we have roughly the right timestamp
if (( times[9-i] < times[8-i] + 20 ))
then
# If the user has gone over the threshold
if (( bytes[9-i] > bytes[8-i] + THRESHOLD2 * 10 ))
then let ++counter
fi
fi
done
# Now check counter
if (( counter == 9 ))
then vzctl stop $veid
fi
# Same for lower threshold
counter=0
for (( i=0; i<3; i++ ))
do
# If we have roughly the right timestamp
if (( times[3-i] < times[2-i] + 20 ))
then
# If the user has gone over the threshold
if (( bytes[3-i] > bytes[2-i] + THRESHOLD1 * 10 ))
then let ++counter
fi
fi
done
# Now check counter
if (( counter == 2 ))
then vzctl stop $veid
fi
done
sleep 10
done
I've checked the numbers in /var/log/outbound_ddos/vm101.log and they're increasing by more than the threshold, but nothing is happening.
I added some echo statements to try and figure out where the problem is and it seems to be this comparison that's returning false:
if (( bytes[9-i] > bytes[8-i] + THRESHOLD2 * 10 ))
So then I tried the following, which printed out nothing:
echo ${bytes[9-i]}
Could anyone point me in the right direction? I think the script is nearly done, probably something very simple.
Your shell runs the while read loop in a subshell (see here for why it does not work as expected), so your array magic does not propagate outside the tail | while construct.
Read this and fix accordingly :-)

Resources