bc: prevent "divide by zero" runtime error on multiple operations - bash

I'm using bc to do a series of calculation.
I'm calling it though a bash script that first of all puts all the expressions to be calculated in a single variable, the passes them to bc to calculate the results.
The script is something like that:
#!/bin/bash
....
list=""
for frml in `cat $frmlList`
do
value=`echo $frml`
list="$list;$value"
done
echo "scale=3;$list"| bc
the frmlList variable contains a list of expressions that are the output of another program, for simplicity i don't mention every operation, but on its contents are done some sed operations before to assign it to the "value" variable.
In the end, the "list" variable contains a list of expressions separated by semicolon that bc understands.
Now what happens is that in my formula list, sometimes happens that there is a division by 0.
When it happens bc stops its computation giving a "Runtime Error: divide by zero".
I would bc to not end its work on that error, but to skip it and continue with the next formula evaluation.
Is possible to achieve something like that?
The same thing happens in a simpler situation:
echo "scale=2;1+1;1/0;2+2;" | bc
the output is
2
Runtime error (func=(main), adr=17): Divide by zero
I would like to have something like
2
Runtime error (func=(main), adr=17): Divide by zero
4
Thank you in advance :)

Ok, in the end i found a workaround that does the trick quite well.
The idea is to parallelize the execution of bc using subshells, this way if an evaluation fails the other can be still done.
In my script i did something like this:
#!/bin/bash
i=0
for frml in `cat $frmlList`
do
i=$((i+1))
(echo "scale=3;$value"| bc -l extensions.bc > "file_$i.tmp") &
if (( $i % 10 == 0 )); then
wait;
fi # Limit to 10 concurrent subshells.
done
#do something with the written files

I know no simple way to do this. If the expressions are independent, you can try to run them all in bc. If that fails, feed them to bc one by one, skipping the broken ones.
If expressions depend on each other, then you probably need something more powerful than bc. Or you can try to append expression after expression to an input file. If bc fails, remove the last one (maybe restore the file from a backup) and try with the next one.

Related

variable passing through awk command [duplicate]

This question already has answers here:
How do I set a variable to the output of a command in Bash?
(15 answers)
Closed 1 year ago.
here's my issue, I have a bunch of fastq.gz files and I need to determine the number of lines of it (this is not the issue), and from that number of line derive a value that determine a threshold used as a variable used down in the same loop. I browsed but cannot find how to do it. here's what I have so far:
for file in *R1.fastq*; do
var=echo $(zcat "$file" | $((`wc -l`/400000)))
for i in *Bacter*; do
awk -v var1=$var '{if($2 >= var1) print $0}' ${i} | wc -l >> bacter-filtered.txt
done
done
I get the error message: -bash: 14850508/400000: No such file or directory
any help would be greatly appreciated !
The problem is in the line
var=echo $(zcat "$file" | $((`wc -l`/400000)))
There are a bunch of shell syntax elements here combined in ways that don't connect up with each other. To keep things straight, I'd recommend splitting it into two separate operations:
lines=$(zcat "$file" | wc -l)
var=$((lines/400000))
(You may also have to do something about the output to bacter-filtered.txt -- it's just going to contain a bunch of numbers, with no identifications of which ones come from which files. Also since it always appends, if you run this twice you'll have the output from both runs stuck together. You might want to replace all those appends with a single > bacter-filtered.txt after the last done, so the whole output just gets stored directly.)
What's wrong with the original? Well, let's start with this:
zcat "$file" | $((`wc -l`/400000))
Unless I completely misunderstand, the purpose here is to extract $file (with zcat), count lines in the result (with wc -l), and divide that by 400000. But since the output of zcat isn't piped directly to wc, it's piped to a complex expression involving wc, it's somewhat ambiguous what should happen, and is actually different under different shells. In zsh, it does something completely different from that: it lets wc read from the script's stdin (generally your Terminal), divides the result from that by 400000, and then pipes the output of zcat to that ... number?
In bash, it does something closer to what you want: wc actually does read from the output of zcat, so the second part of the pipe essentially turns into:
... | $((14850508/400000))
Now, what I'd expect to happen at this point (and happens in my tests) is that it should evaluate $((14850508/400000)) into 37, giving:
... | 37
which will then try to execute 37 as a command (because it's part of a pipeline, and therefore is supposed to be a command). But for some reason it's apparently not evaluating the division and just trying to execute 14850508/400000 as a command. Which doesn't really work any better or worse than 37, so I guess it doesn't matter much.
So that's where the error is coming from, but there's actually another layer of confusion in the original line. Suppose that internal pipeline was fixed so that it properly output "37" (rather than trying to execute it). The outer structure would then be:
var=echo $(cmdthatprints37)
The $( ) basically means "run the command inside, and substitute its output into the command line here", so that would evaluate to:
var=echo 37
...which, in shell syntax, means "run the command 37 with var set to "echo" in its environment.
The solution here would be simple. The echo is messing everything up so remove it:
var=$(cmdthatprints37)
...which evaluates to:
var=37
...which is what you want. Except that, as I said above, it'd be better to split it up and do the command bits and the math separately rather than getting them mixed up.
BTW, I'd also recommend some additional double-quoting of shell variables; shellcheck.net will be happy to point out where.

Trying to create a bash script that adds positive integers taken from input on a single line with spaces

Im trying to take input from the terminal one line with spaces in between for example input would be something like this.
-1 -1 1
then im trying to take that input and add them together.
the catch is im only supposed to add positive integers so i need to remove the dashes.
I have
read Str
str=( $Str );
arr=${str//-/}
echo "$((${arr[#]/%/+}0))"
it seems like its only removing one instance of the dash and not the rest. Not sure which direction to take. Im sure there are multiple solutions.
Any help would be appreciated. I was also thinking maybe an If statement that could remove dashes before adding but not sure how exactly to even begin that.
If you just want to accumulate the words one at a time into a sum, removing any optional leading - characters, you can do that with something like:
read line
((sum = 0))
for num in ${line} ; do
absnum=${num//-/}
((sum += absnum))
done
echo "Sum is" ${sum}
Your method of removing leading negative sign is sound, but you don't really need to create an array to do the work. You can just iterate over the words as shown.
Keep in mind a real program should be a little more robust, handling non-numerics and such without falling in a screaming heap. You could do something like this immediately after the assignment to absnum:
[[ ! "${absnum}" =~ ^[1-9][0-9]*|0$ ]] && echo Bad number ${num} && exit 1
This would ensure it was a valid non-negative integer. A sample run follows so you can adjust to whatever test data you'd like to use:
pax> echo '3 0 --1 -2 4 40' | bash myscript.bash
Sum is 50

performing arithmetic on variables set by while-read loop

I want to perform some arithmetic and floating point arithmetic operations on some variables that I read in from a while loop. My code looks like this:
while read n p qlen slen len; do
random_var=$(( $qlen + 0 )) # gives all zeros, not saving qlen
qcov=$(echo $len / $qlen | bc -l) #len and qlen are nothing, not working
qcov=$( bc <<< "scale=3; $len / $qlen" ) #same reason as above, not working
done < input_file.txt
I am pulling each line in input_file.txt and parsing the data into their respective variable names, n, p, qlen, slen, and len. The input file input_file.txt looks like this:
test1 12.345 123 234 12
test2 23.456 345 678 43
test3 98.765 6537 874 346
...
The problem I am having is when I try to perform some arithmetic operation on any of the variables from the read. I can echo the variables just fine, but as soon as I try to perform any kind of arithmetic manipulation, the values are nothing. I suspect that they are not numerals to begin with, some sort of string type (?). I've tried many integer and floating point arithmetic statements and can't seem to preserve the values in the variables. bc commands, arithmetic expansion, echo piped into bc, the triple <<< for bc, all of these solutions I've found online do not work.
The most common errors I receive are:
strings of nothing
(system-in) error 1: parse error
(system-in) error 2: parse error
I've thought about using an awk but I need to perform these operations on every line, and awk keeps going until the end of the file. Is there still a way to do what I want using an awk? I don't really know.
If anyone has an answer or an alternative way to pull data out of the file, I'd really appreciate it.
Thanks in advance.
UPDATE:
some fiddling around with the code has made me realize that everything in the input_file.txt is being saved into only the first variable n. This is why the other variables are empty and why the echo of all the variables looked correct.
Any ideas why the values are not being saved into the correct variables?
I don't know what you really want to obtain. but it seems something like
awk '{print $5/$3}' input_file.txt
0.097561
0.124638
0.0529295

Read range of numbers into a for loop

So, I am building a bash script which iterates through folders named by numbers from 1 to 9. The script depends on getting the folder names by user input. My intention is to use a for loop using read input to get a folder name or a range of folder names and then do some stuff.
Example:
Let's assume I want to make a backup with rsync -a of a certain range of folders. Usually I would do:
for p in {1..7}; do
rsync -a $p/* backup.$p
done
The above would recursively backup all content in the directories 1 2 3 4 5 6 and 7 and put them into folders named as 'backup.{index-number}'. It wouldn't catch folders/files with a leading . but that is not important right now.
Now I have a similar loop in an interactive bash script. I am using select and case statements for this task. One of the options in case is this loop and it shall somehow get a range of numbers from user input. This now becomes a problem.
Problem:
If I use read to get the range then it fails when using {1..7} as input. The input is taken literally and the output is just:
{1..7}
I really would like to know why this happens. Let me use a more descriptive example with a simple echo command.
var={1..7} # fails and just outputs {1..7}
for p in $var; do echo $p;done
read var # Same result as above. Just outputs {1..7}
for p in $var; do echo $p;done
for p in {1..7}; do echo $p;done # works fine and outputs the numbers 1-7 seperated with a newline.
I've found a workaround by storing the numbers in an array. The user can then input folder names seperated by a space character like this: 1 2 3 4 5 6 7
read -a var # In this case the output is similar to the 3rd loop above
for p in ${var[#]}; do echo $p; done
This could be a way to go but when backing up 40 folders ranging from 1-40 then adding all the numbers one-by-one completely makes my script redundant. One could find a solution to one of the millennium problems in the same time.
Is there any way to read a range of numbers like {1..9} or could there be another way to get input from terminal into the script so I can iterate through the range within a for-loop?
This sounds like a question for google but I am obviously using the wrong patterns to get a useful answer. Most of similar looking issues on SO refer to brace and parameter expansion issues but this is not exactly the problem I have. However, to me it feels like the answer to this problem is going in a similar direction. I fail to understand why when a for-loop for assigning {1..7} to a variable works but doing the same like var={1..7} doesn't. Plz help -.-
EDIT: My bash version:
$ echo $BASH_VERSION
4.2.25(1)-release
EDIT2: The versatility of a brace expansion is very important to me. A possible solution should include the ability to define as many ranges as possible. Like I would like to be able to choose between backing up just 1 folder or a fixed range between f.ex 4-22 and even multiple options like folders 1,2,5,6-7
Brace expansion is not performed on the right-hand side of a variable, or on parameter expansion. Use a C-style for loop, with the user inputing the upper end of the range if necessary.
read upper
for ((i=1; i<=$upper; i++)); do
To input both a lower and upper bound separated by whitespace
read lower upper
for (i=$lower; i <= $upper; i++)); do
For an arbitrary set of values, just push the burden to the user to generate the appropriate list; don't try to implement your own parser to process something like 1,2,20-22:
while read p; do
rsync -a $p/* backup.$p
done
The input is one value per line, such as
1
2
20
21
22
Even if the user is using the shell, they can call your script with something like
printf '%s\n' 1 2 20..22 | backup.sh
It's easier for the user to generate the list than it is for you to safely parse a string describing the list.
The evil eval
$ var={1..7}
$ for i in $(eval echo $var); do echo $i; done
this also works,
$ var="1 2 {5..9}"
$ for i in $(eval echo $var); do echo $i; done
1
2
5
6
7
8
9
evil eval was a joke, that is, as long as you know what you're evaluating.
Or, with awk
$ echo "1 2 5-9 22-25" |
awk -v RS=' ' '/-/{split($0,a,"-"); while(a[1]<=a[2]) print a[1]++; next}1'
1
2
5
6
7
8
9
22
23
24
25

Bash script using negative to cycle through

I'm trying to run a bash script that includes a nested for loop within which a variable should cycle through negative exponents, viz:
for ABPOW in {-11..-9}
do
ABC = $((10**$ABPOW))
for ABCOEFF in {1..9}
do
sed -e 's/ACOEFF/'$ABC'/'\
This is only the inner two for loops of the code. When the values in the first bracket (for ABPOW) are positive, the code runs fine. However, when I have them as i do above, which is what I need, the error communicated to screen is:
./scripting_test2.bash: line 30: 10**-11: exponent less than 0 (error token is "1")
How do I make this run? Thanks in advance.
PS: I tried putting a negative sign in front of $ABPOW but the exponents are still recorded as positive.
Bash does not support floating point arithmetic (which is necessary for raising something into a negative power). Instead, you should use the bc utility.
ABC=$(bc -l <<< "10 ^($ABPOW)")
Also, there should be no spaces before and after the = in variable assignments
Here's another way:
perl -e 'for $abpow (-11..-9) { $abc=10**$abpow; for (1..9) { system("echo $abc"); } }'
Bash doesn't support floating point data types.
$ man bash|grep -c -i float
0
Do you have python installed?
ABC=$( python -c "print 10**$ABPOW" )
One alternative is to use awk for your complete script that has full support for floating point arithmetic:
awk 'BEGIN{for(i=-11; i<=-9; i++) {ABC=10^i; print ABC}}'
1e-11
1e-10
1e-09

Resources