How to assign multiple string values into a variable in shell scripts? - shell

I'm trying to loop a text file changing the number and the hole string is store into a shell variable, to print it latter
I've tried echo the string an the integer (number from the loop) using ">>"
#!/bin/bash
a=0
while [ "$a" -lt 4 ]
do
echo '<div name="block-'${a}'">' >> $sub_main
((a++))
done
echo "done"
echo $sub_main
The output from the script should be:
<div name="block-0">
<div name="block-1">
<div name="block-2">
<div name="block-3">

>> appends the out putto a file. If you want to append to a variable you can use an assignment with the variable to append to at the beginning of the right value.
a=0;
sub_main="";
while [ "${a}" -lt 4 ]; do
sub_main="${sub_main}\n<div name=\"block-${a}\">";
((a++));
done;
echo "done";
echo "${sub_main}";
Or, if the appending to the file was intended, you can simply use cat instead of echo, to output the file's contents.
I.e. replace
echo $sub_main
with:
cat "${sub_main}";

Use command substitution to capture the output into a variable:
#!/bin/bash
sub_main=$(
for (( a = 0; a < 4; a++ )); do
printf '<div name="block-%d">\n' "$a"
done
)
# ... do other stuff ...
echo "$sub_main"
Or you might want to use a function to defer execution until later:
#!/bin/bash
sub_main() {
for (( a = 0; a < 4; a++ )); do
printf '<div name="block-%d">\n' "$a"
done
}
# ... do other stuff ...
sub_main

Related

Exporting excess arguments of a bash file to another text file

I'm trying to write a bash file which normally takes 3 arguments normally. There is an extra condition which is invoked when there are more than 3 arguments given. This condition should export the extra arguments to a text file called excess. This is the code I have so far:
if [ $# -gt 3]; then
for ((i = 4; i <= $#; i++)); do
echo "$i" >> "excess.txt"
done
fi
Instead of exporting the actual arguments, the loop is exporting the numbers 4,5,6... to the text file.
I'm not quite sure why this is happening seeing that I've used the dollar sign before 'i' in the echo statement.
You should use indirect expansion ${!}:
for ((i = 4; i <= $#; i++)); do
echo "${!i}" >> "excess.txt"
done
also, you can do
shift 3; echo "$#" >> execss.txt
You can do this with one line:
printf '%s\n' "${#:4}" > excess.txt
"${#:4}" expands to "$4" "$5" "$6" ..., and the printf command has its own loop to output its format string once per argument.

How do I create large CSVs in seconds?

I am trying to create 1000s of large CSVs rapidly. This function generates the CSVs:
function csvGenerator () {
for ((i=1; i<=$NUMCSVS; i++)); do
CSVNAME=$DIRNAME"-"$CSVPREFIX$i$CSVEXT
HEADERARRAY=()
if [[ ! -e $CSVNAME ]]; then #Only create csv file if it not exist
touch $CSVNAME
echo "file: "$CSVNAME "created at $(date)" >> ../status.txt
fi
for ((j=1; j<=$NUMCOLS; j++)); do
if (( j < $NUMCOLS )) ; then
HEADERNAME=$DIRNAME"-csv-"$i"-header-"$j", "
elif (( j == $NUMCOLS )) ; then
HEADERNAME=$DIRNAME"-csv-"$i"-header-"$j
fi
HEADERARRAY+=$HEADERNAME
done
echo $HEADERARRAY > $CSVNAME
for ((k=1; k<=$NUMROWS; k++)); do
ROWARRAY=()
for ((l=1; l<=$NUMCOLS; l++)); do
if (( l < $NUMCOLS )) ; then
ROWVALUE=$DIRNAME"-csv-"$i"-r"$k"c"$l", "
elif (( l == $NUMCOLS )) ; then
ROWVALUE=$DIRNAME"-csv-"$i"-r"$k"c"$l
fi
ROWARRAY+=$ROWVALUE
done
echo $ROWARRAY >> $CSVNAME
done
done
}
The script takes ~3 mins to generate a CSV with 100k rows and 70 cols. What do I need to do to generate these CSVs at the rate of 1 CSV/~10 seconds?
Let me start by saying that bash and "performant" don't usually go together in the same sentence. As other commentators suggested, awk may be a good choice that's adjacent in some senses.
I haven't yet had a chance to run your code, but it opens and closes the output file once per row — in this example, 100,000 times. Each time it must seek to the end of the file so that it can append the latest row.
Try pulling the actual generation (everything after for ((j=1; j<=$NUMCOLS; j++)); do) into a new function, like generateCsvContents. In that new function, don't reference $CSVNAME, and remove the redirections on the echo statements. Then, in the original function, call the new function and redirect its output to the filename. Roughly:
function csvGenerator () {
for ((i=1; i<=NUMCSVS; i++)); do
CSVNAME=$DIRNAME"-"$CSVPREFIX$i$CSVEXT
if [[ ! -e $CSVNAME ]]; then #Only create csv file if it not exist
echo "file: $CSVNAME created at $(date)" >> ../status.txt
fi
# This will create $CSVNAME if it doesn't yet exist
generateCsvContents > "$CSVNAME"
done
}
function generateCsvContents() {
HEADERARRAY=()
for ((j=1; j<=NUMCOLS; j++)); do
if (( j < NUMCOLS )) ; then
HEADERNAME=$DIRNAME"-csv-"$i"-header-"$j", "
elif (( j == NUMCOLS )) ; then
HEADERNAME=$DIRNAME"-csv-"$i"-header-"$j
fi
HEADERARRAY+=$HEADERNAME
done
echo $HEADERARRAY
for ((k=1; k<=NUMROWS; k++)); do
ROWARRAY=()
for ((l=1; l<=NUMCOLS; l++)); do
if (( l < NUMCOLS )) ; then
ROWVALUE=$DIRNAME"-csv-"$i"-r"$k"c"$l", "
elif (( l == NUMCOLS )) ; then
ROWVALUE=$DIRNAME"-csv-"$i"-r"$k"c"$l
fi
ROWARRAY+=$ROWVALUE
done
echo "$ROWARRAY"
done
}
"Not this way" is I think the answer.
There are a few problems here.
You're not using your arrays as arrays. When you treat them like strings, you affect only the first element in the array, which is misleading.
The way you're using >> causes the output file to be opened and closed once for every line. That's potentially wasteful.
You're not quoting your variables. In fact, you're quoting the stuff that doesn't need quoting, and not quoting the stuff that does.
Upper case variable names are not recommended, due to the risk of collision with system variables. ref
Bash isn't good at this. Really.
A cleaned up version of your function might look like this:
csvGenerator2() {
for (( i=1; i<=NUMCSVS; i++ )); do
CSVNAME="$DIRNAME-$CSVPREFIX$i$CSVEXT"
# Only create csv file if it not exist
[[ -e "$CSVNAME" ]] && continue
touch "$CSVNAME"
date "+[%F %T] created: $CSVNAME" | tee -a status.txt >&2
HEADER=""
for (( j=1; j<=NUMCOLS; j++ )); do
printf -v HEADER '%s, %s-csv-%s-header-%s' "$HEADER" "$DIRNAME" "$i" "$j"
done
echo "${HEADER#, }" > "$CSVNAME"
for (( k=1; k<=NUMROWS; k++ )); do
ROW=""
for (( l=1; l<=NUMCOLS; l++ )); do
printf -v ROW '%s, %s-csv-%s-r%sc%s' "$ROW" "$DIRNAME" "$i" "$k" "$l"
done
echo "${ROW#, }"
done >> "$CSVNAME"
done
}
(Note that I haven't switched the variables to lower case because I'm lazy, but it's still a good idea.)
And if you were to make something functionally equivalent in awk:
csvGenerator3() {
awk -v NUMCSVS="$NUMCSVS" -v NUMCOLS="$NUMCOLS" -v NUMROWS="$NUMROWS" -v DIRNAME="$DIRNAME" -v CSVPREFIX="$CSVPREFIX" -v CSVEXT="$CSVEXT" '
BEGIN {
for ( i=1; i<=NUMCSVS; i++) {
out=sprintf("%s-%s%s%s", DIRNAME, CSVPREFIX, i, CSVEXT)
if (!system("test -e " CSVNAME)) continue
system("date '\''+[%F %T] created: " out "'\'' | tee -a status.txt >&2")
comma=""
for ( j=1; j<=NUMCOLS; j++ ) {
printf "%s%s-csv-%s-header-%s", comma, DIRNAME, i, j > out
comma=", "
}
printf "\n" >> out
for ( k=1; k<=NUMROWS; k++ ) {
comma=""
for ( l=1; l<=NUMCOLS; l++ ) {
printf "%s%s-csv-%s-r%sc%s", comma, DIRNAME, i, k, l >> out
comma=", "
}
printf "\n" >> out
}
}
}
'
}
Note that awk does not suffer from the same open/closer overhead mentioned earlier with bash; when a file is used for output or as a pipe, it gets opened once and is left open until it is closed.
Comparing the two really highlights the choice you need to make:
$ time bash -c '. file; NUMCSVS=1 NUMCOLS=10 NUMROWS=100000 DIRNAME=2 CSVPREFIX=x CSVEXT=.csv csvGenerator2'
[2019-03-29 23:57:26] created: 2-x1.csv
real 0m30.260s
user 0m28.012s
sys 0m1.395s
$ time bash -c '. file; NUMCSVS=1 NUMCOLS=10 NUMROWS=100000 DIRNAME=3 CSVPREFIX=x CSVEXT=.csv csvGenerator3'
[2019-03-29 23:58:23] created: 3-x1.csv
real 0m4.994s
user 0m3.297s
sys 0m1.639s
Note that even my optimized bash version is only a little faster than your original code.
Refactoring your two inner for-loops to loops like this will save time:
for ((j=1; j<$NUMCOLS; ++j)); do
HEADERARRAY+=$DIRNAME"-csv-"$i"-header-"$j", "
done
HEADERARRAY+=$DIRNAME"-csv-"$i"-header-"$NUMCOLS

Echo a variable with an index at the end in bash script

I am trying to allocate a bunch of temp files in a loop and export them from within the loop. I then want to loop thru again and echo the values.
for (( i=1; i<=5; i++ ))
do
dd if=/dev/zero of=/tmp/mh1_$i.out bs=1024 count=1024 status=none
declare TEMP_FILE_${i}="/tmp/mh1_${i}.out"
export "TEMP_FILE_${i}"
done
If I do a echo $TEMP_FILE_1 it correctly prints /tmp/mh1_1.out
But when I try this in a loop it prints 1, 2, 3, 4 and 5.
for (( i=1; i<=5; i++ ))
do
echo $TEMP_FILE_${i} --> This prints the i value instead of /tmp/mh1_x.out
done
How do I escape the index $i in the echo above to see the real file name ?
I suggest using the mktemp utility with arrays:
# create tmp files
tmp_files=()
for ((i=0;i<5;++i)); do
tmp_files+=("$(mktemp --tmpdir mh1_XXXXXX.out)") || exit 1
done
# process all tmp files
for file in "${tmp_files[#]}"; do
echo "$file"
# do something with the file ...
done
# list all tmp files
printf '%s\n' "${tmp_files[#]}"
# list 2nd tmp file
echo "${tmp_files[1]}"
In order to make your code work, you need to use variable indirection and change your second for loop to:
for ((i=1;i<=5;++i)); do
var=temp_file_$i
echo "${!var}"
done
Don't use uppercase variables as they could clash with environmental or internal shell variables. Also, note that export is needed only when you want to pass the variables to child processes in the environment.
Why not use bash arrays?
export temp_file
loop
temp_file[$i]="/tmp/mh1_${i}.out"
...
endloop
Then loop over the array

bash substitution call that increments a variable

I'm trying to define a bash function returning an incremented id
that I can access directly using bash substitution:
#!/bin/bash
getId() {
echo "$x"
x=$((x+1))
}
x=0
echo "id1: $(getId)"
echo "id2: $(getId)"
However the variable is not incremented and I cannot figure out why.
id1: 0
id2: 0
Please, does someone have an explanation for this behaviour?
getId() {
echo "$x"
((x++))
}
x=0
echo -n "id1: "
getId
echo -n "id2: "
getId
Output:
id1: 0
id2: 1
There is no easy way I know of to do it in a sub-shell call using the syntax you have (in the echo line).
An alternate would be:
#!/bin/bash
export x=0
incId() {
#echo "$x"
(( x += 1))
}
incId
echo "id1: $x"
incId
echo "id2: $x"
But here you need the out-of-the-echo-line incId function call to get the id incremented.
It also starts counting from 1, not 0.
Using the let shell command is the better way to do math too.
Using (( ... )) is the right way to do shell arithmetic
Might as well make it generic:
incr() { (( $1 += ${2:-1} )); }
Examples:
incr x ; echo $x # => 1
incr x ; echo $x # => 2
incr x 4; echo $x # => 6
incr x -2; echo $x # => 4

Passing array to function of shell script

How to pass array as function in shell script?
I written following code:
function test(){
param1 = $1
param2 = $2
for i in ${$param1[#]}
do
for j in ${param2[#]}
do
if($(i) = $(j) )
then
echo $(i)
echo $(j)
fi
done
done
}
but I am getting line 1: ${$(param1)[#]}: bad substitution
There are multiple problems:
you can't have spaces around the = when assigning variables
your if statement has the wrong syntax
array passing isn't right
try not to call your function test because that is a shell command
Here is the fixed version:
myFunction(){
param1=("${!1}")
param2=("${!2}")
for i in ${param1[#]}
do
for j in ${param2[#]}
do
if [ "${i}" == "${j}" ]
then
echo ${i}
echo ${j}
fi
done
done
}
a=(foo bar baz)
b=(foo bar qux)
myFunction a[#] b[#]
You can use the following script accordingly
#!/bin/bash
param[0]=$1
param[1]=$2
function print_array {
array_name=$1
eval echo \${$array_name[*]}
return
}
print_array param
exit 0
A simple way :
function iterate
{
n=${#detective[#]}
for (( i=0; i<n; i++ ))
do
echo ${detective[$i]}
done
}
detective=("Feluda" "Sharlockhomes" "Bomkesh" )
iterate ${detective[#]}

Resources