I want to perform some arithmetic and floating point arithmetic operations on some variables that I read in from a while loop. My code looks like this:
while read n p qlen slen len; do
random_var=$(( $qlen + 0 )) # gives all zeros, not saving qlen
qcov=$(echo $len / $qlen | bc -l) #len and qlen are nothing, not working
qcov=$( bc <<< "scale=3; $len / $qlen" ) #same reason as above, not working
done < input_file.txt
I am pulling each line in input_file.txt and parsing the data into their respective variable names, n, p, qlen, slen, and len. The input file input_file.txt looks like this:
test1 12.345 123 234 12
test2 23.456 345 678 43
test3 98.765 6537 874 346
...
The problem I am having is when I try to perform some arithmetic operation on any of the variables from the read. I can echo the variables just fine, but as soon as I try to perform any kind of arithmetic manipulation, the values are nothing. I suspect that they are not numerals to begin with, some sort of string type (?). I've tried many integer and floating point arithmetic statements and can't seem to preserve the values in the variables. bc commands, arithmetic expansion, echo piped into bc, the triple <<< for bc, all of these solutions I've found online do not work.
The most common errors I receive are:
strings of nothing
(system-in) error 1: parse error
(system-in) error 2: parse error
I've thought about using an awk but I need to perform these operations on every line, and awk keeps going until the end of the file. Is there still a way to do what I want using an awk? I don't really know.
If anyone has an answer or an alternative way to pull data out of the file, I'd really appreciate it.
Thanks in advance.
UPDATE:
some fiddling around with the code has made me realize that everything in the input_file.txt is being saved into only the first variable n. This is why the other variables are empty and why the echo of all the variables looked correct.
Any ideas why the values are not being saved into the correct variables?
I don't know what you really want to obtain. but it seems something like
awk '{print $5/$3}' input_file.txt
0.097561
0.124638
0.0529295
Related
This question already has answers here:
How do I set a variable to the output of a command in Bash?
(15 answers)
Closed 1 year ago.
here's my issue, I have a bunch of fastq.gz files and I need to determine the number of lines of it (this is not the issue), and from that number of line derive a value that determine a threshold used as a variable used down in the same loop. I browsed but cannot find how to do it. here's what I have so far:
for file in *R1.fastq*; do
var=echo $(zcat "$file" | $((`wc -l`/400000)))
for i in *Bacter*; do
awk -v var1=$var '{if($2 >= var1) print $0}' ${i} | wc -l >> bacter-filtered.txt
done
done
I get the error message: -bash: 14850508/400000: No such file or directory
any help would be greatly appreciated !
The problem is in the line
var=echo $(zcat "$file" | $((`wc -l`/400000)))
There are a bunch of shell syntax elements here combined in ways that don't connect up with each other. To keep things straight, I'd recommend splitting it into two separate operations:
lines=$(zcat "$file" | wc -l)
var=$((lines/400000))
(You may also have to do something about the output to bacter-filtered.txt -- it's just going to contain a bunch of numbers, with no identifications of which ones come from which files. Also since it always appends, if you run this twice you'll have the output from both runs stuck together. You might want to replace all those appends with a single > bacter-filtered.txt after the last done, so the whole output just gets stored directly.)
What's wrong with the original? Well, let's start with this:
zcat "$file" | $((`wc -l`/400000))
Unless I completely misunderstand, the purpose here is to extract $file (with zcat), count lines in the result (with wc -l), and divide that by 400000. But since the output of zcat isn't piped directly to wc, it's piped to a complex expression involving wc, it's somewhat ambiguous what should happen, and is actually different under different shells. In zsh, it does something completely different from that: it lets wc read from the script's stdin (generally your Terminal), divides the result from that by 400000, and then pipes the output of zcat to that ... number?
In bash, it does something closer to what you want: wc actually does read from the output of zcat, so the second part of the pipe essentially turns into:
... | $((14850508/400000))
Now, what I'd expect to happen at this point (and happens in my tests) is that it should evaluate $((14850508/400000)) into 37, giving:
... | 37
which will then try to execute 37 as a command (because it's part of a pipeline, and therefore is supposed to be a command). But for some reason it's apparently not evaluating the division and just trying to execute 14850508/400000 as a command. Which doesn't really work any better or worse than 37, so I guess it doesn't matter much.
So that's where the error is coming from, but there's actually another layer of confusion in the original line. Suppose that internal pipeline was fixed so that it properly output "37" (rather than trying to execute it). The outer structure would then be:
var=echo $(cmdthatprints37)
The $( ) basically means "run the command inside, and substitute its output into the command line here", so that would evaluate to:
var=echo 37
...which, in shell syntax, means "run the command 37 with var set to "echo" in its environment.
The solution here would be simple. The echo is messing everything up so remove it:
var=$(cmdthatprints37)
...which evaluates to:
var=37
...which is what you want. Except that, as I said above, it'd be better to split it up and do the command bits and the math separately rather than getting them mixed up.
BTW, I'd also recommend some additional double-quoting of shell variables; shellcheck.net will be happy to point out where.
This question already has answers here:
How do I iterate over a range of numbers defined by variables in Bash?
(20 answers)
Closed 3 years ago.
I am still learning how to shell script and I have been given a challenge to make it easier for me to echo "Name1" "Name2"..."Name15" and I'm not too sure where to start, I've had ideas but I don't want to look silly if I mess it up. Any help?
I haven't actually tried anything just yet it's all just been mostly thought.
#This is what I wrote to start
#!/bin/bash
echo "Name1"
echo "Name2"
echo "Name3"
echo "Name4"
echo "Name5"
echo "Name6"
echo "Name7"
echo "Name8"
echo "Name9"
echo "Name10"
echo "Name11"
echo "Name12"
echo "Name13"
echo "Name14"
echo "Name15"
My expected results are obviously just for it to output "Name1" "Name2" etc. But I'm looking for a more creative way to do it. If possible throw in a few ways to do it so I can learn. Thank you.
The easiest (possibly not the most creative) way to do this is to use printf:
printf "%s\n" name{1..15}
This relies on bash brace expansion {1..15} to have the 15 strings.
Use a for loop
for i in {1..15};do echo "Name$i";done
A few esoteric solutions, from the least to the most unreasonable :
base64 encoded string :
base64 -d <<<TmFtZTEKTmFtZTIKTmFtZTMKTmFtZTQKTmFtZTUKTmFtZTYKTmFtZTcKTmFtZTgKTmFtZTkKTmFtZTEwCk5hbWUxMQpOYW1lMTIKTmFtZTEzCk5hbWUxNApOYW1lMTUK
The weird chain is your expected result encoded in base64, an encoding generally used to represent binary data as text. base64 -d <<< weirdChain is passing the weird chain as input to the base64 tool and asking it to decode it, which displays your expected result
generate an infinite stream of "Name", truncate it, use line numbers :
yes Name | awk 'NR == 16 { exit } { printf("%s%s\n", $0, NR) }'
yes outputs an infinite stream of what it's passed as argument (or y by default, used to automatize interactive scripts asking for [y/n] confirmation). The awk command exits once it reaches the 16th line, and otherwise prints its input (provided by yes) followed by the line number. The truncature could as easily be done with head -15, and I've tried using the nl "number line" utility or grep -n to number lines, but they always added the line numbers as prefix which required an extra re-formatting step.
read random binary data and hope to stumble on all the lines you want to output :
timeout 1d strings /dev/urandom | grep -Eo "Name(1[0-5]|[1-9])" | sort -uV
strings /dev/urandom will extract ascii sequences from the binary random source /dev/urandom, grep will filter those which respect the format of a line of your expected output and sort will reorder those lines in the correct order. Since sort needs to have a received its whole input before it reorders it and /dev/urandom won't stop producing data, we use timeout 1d to stop reading from /dev/urandom after a whole day in hope it has sifted through enough random data to find your 15 lines (I'm not sure that's even remotely likely).
use an HTTP client to retrieve this page, extract the bash script you posted and execute it.
my_old_script=$(curl "https://stackoverflow.com/questions/57818680/" | grep "#This is what I wrote to start" -A 18 | tail -n+4)
eval "$my_old_script"
curl is a command line tool that can be used as an HTTP client, grep with its -A 18 parameter will select the "This is what I wrote to start" text and the 18 lines that follow, tail will remove the first 3 lines, and eval will execute your script.
While it will be much more efficient than the previous solution, it's an even less reasonable solution because high-rep users can edit your question to make this solution execute arbitrary code on your computer. Ideally you'd be using an HTML-aware parser rather than basic string manipulation to extract the code, but we're not talking about best practices here...
I am writing a bash script that loops over a large file of data which i have extracted the key parts I need to use. It seems quite trivial when I was trying to do it but all I need to do is something akin to,
string1=...
string2=...
correct=0
for i in 1..29
do
if [string1[i] == string2[i]]
then
correct=correct+1
fi
done
When I tried doing something like this I get a Bad Substitution which I assume is because some of the key's look like this,
`41213343323455122411141331555 - key`
`3113314233111 22321112433111* - answer`
The spaces and occational * that are found don't need special treatment in my case, just a simple comparison of each index.
#!/bin/bash
answersCorrect=0
for i in $(nawk 'BEGIN{ for(i=1;i<=29;i++) print i}')
do
if [ "${answer:i:1}" = "${key:i:1}" ]
then
answersCorrect=$answersCorrect+1 #this line#
fi
done
I am getting no compiler errors now however I don't think i'm incrementing answersCorrect correctly. When I output it it is something like 0+1+1+1 instead of just 3 (this segment is being used inside a while loop)
Fixed Solution for that line : answersCorrect=$((answersCorrect+1))
The original problem is fixed by comments and some extra work of #Mikel.
An alternative is comparing the strings after converting the strings to lines.
diff --suppress-common-lines <(fold -w1 <<< "${string1}") <(fold -w1 <<< "${string2}") |
grep -c "^<"
I'm using bc to do a series of calculation.
I'm calling it though a bash script that first of all puts all the expressions to be calculated in a single variable, the passes them to bc to calculate the results.
The script is something like that:
#!/bin/bash
....
list=""
for frml in `cat $frmlList`
do
value=`echo $frml`
list="$list;$value"
done
echo "scale=3;$list"| bc
the frmlList variable contains a list of expressions that are the output of another program, for simplicity i don't mention every operation, but on its contents are done some sed operations before to assign it to the "value" variable.
In the end, the "list" variable contains a list of expressions separated by semicolon that bc understands.
Now what happens is that in my formula list, sometimes happens that there is a division by 0.
When it happens bc stops its computation giving a "Runtime Error: divide by zero".
I would bc to not end its work on that error, but to skip it and continue with the next formula evaluation.
Is possible to achieve something like that?
The same thing happens in a simpler situation:
echo "scale=2;1+1;1/0;2+2;" | bc
the output is
2
Runtime error (func=(main), adr=17): Divide by zero
I would like to have something like
2
Runtime error (func=(main), adr=17): Divide by zero
4
Thank you in advance :)
Ok, in the end i found a workaround that does the trick quite well.
The idea is to parallelize the execution of bc using subshells, this way if an evaluation fails the other can be still done.
In my script i did something like this:
#!/bin/bash
i=0
for frml in `cat $frmlList`
do
i=$((i+1))
(echo "scale=3;$value"| bc -l extensions.bc > "file_$i.tmp") &
if (( $i % 10 == 0 )); then
wait;
fi # Limit to 10 concurrent subshells.
done
#do something with the written files
I know no simple way to do this. If the expressions are independent, you can try to run them all in bc. If that fails, feed them to bc one by one, skipping the broken ones.
If expressions depend on each other, then you probably need something more powerful than bc. Or you can try to append expression after expression to an input file. If bc fails, remove the last one (maybe restore the file from a backup) and try with the next one.
I need some help with a bash mathematical expression script. I need the expression below to be a negative 2 to the power of 63 but I have tried all sorts of combinations from "", '', (()) and even keeping the positive value but multiplying it by -1 but I can't seem to get the syntax right.
Original expression:
kw=`expr 2^63 | bc`
gives me 9223372036854775808 (which is correct) but I need to get a -9223372036854775808.
I want to be able to do a kw=expr -2^63 | bc to get a negative results but it's not as straight-forward as I'd hoped. Hence my numerous attempts of different permutations to the expression.
Any help will be very appreciated.
Here you go:
$ kw=$(echo -2^63 | bc)
$ echo $kw
-9223372036854775808
UPDATE
#DigitalTrauma is right, if you're in bash, then using a bash here string is better (one less process, more efficient):
kw=$(bc <<< -2^63)
Since this is bash, you don't even need the echo; you can use a bash here string instead:
$ kw=$(bc <<< -2^63)
$ echo $kw
-9223372036854775808