I am extracting user ratings after making some calculations. If calculated rating exceeds 5.0 (say 5.001), I want to clip it to 5 and if it falls below 1 (say 0.001), I want to limit it to 1.
I will be grateful if there can be a bash script that makes this clipping. Or maybe I can use a python script from command line which takes dollar variable from bash.
You can use awk:
awk '$0<1{$0=1}$0>5{$0=5}1' <<<"$var"
If the variable is less than 1, change it to 1. If it is greater than 5, change it to 5. 1 at the end is always true, so awk performs the default action, which is to print the record.
Some additional explanation:
<<< is an alternative way of writing echo "$var" | awk ..., supported by bash
$0 refers to the current "record", which in this case is the variable that has been echoed to bash
awk works like condition { action }, so each block is only run if the condition before it is true
if no {action} part is specified, the default action is {print}, so that's what the 1 at the end does.
For example:
$ var=0.001
$ awk '$0<1{$0=1}$0>5{$0=5}1' <<<"$var"
1
$ var=5.001
$ awk '$0<1{$0=1}$0>5{$0=5}1' <<<"$var"
5
To overwrite the variable, you can use command substitution:
var=$(awk '$0<1{$0=1}$0>5{$0=5}1' <<<"$var")
If you like confusing yourself, you can do the same thing using two ternary operators instead:
awk '{$0=$0<1?1:$0>5?5:$0}1' <<<"$var"
Related
cat TEXT | awk -v var=$i -v varB=$j '$1~var , $1~varB {print $1}' > PROBLEM HERE
I am passing two variables from an array to parse a very large text file by range. And it works, kind of.
if I use ">" the output to the file will ONLY be the last three lines as verified by cat and a text editor.
if I use ">>" the output to the file will include one complete read of TEXT and then it will divide the second read into the ranges I want.
if I let the output go through to the shell I get the same problem as above.
Question:
It appears awk is reading every line and printing it. Then it goes back and selects the ranges from the TEXT file. It does not do this if I use constants in the range pattern search.
I undestand awk must read all lines to find the ranges I request.
why is it printing the entire document?
How can I get it to ONLY print the ranges selected?
This is the last hurdle in a big project and I am beating my head against the table.
Thanks!
give this a try, you didn't assign varB in right way:
yours: awk -v var="$i" -varB="$j" ...
mine : awk -v var="$i" -v varB="$j" ...
^^
Aside from the typo, you can't use variables in //, instead you have to specify with regular ~ match. Also quote your shell variables (here is not needed obviously, but to set an example). For example
seq 1 10 | awk -v b="3" -v e="5" '$0 ~ b, $0 ~ e'
should print 3..5 as expected
It sounds like this is what you want:
awk -v var="foo" -v varB="bar" '$1~var{f=1} f{print $1} $1~varB{f=0}' file
e.g.
$ cat file
1
2
foo
3
4
bar
5
foo
6
bar
7
$ awk -v var="foo" -v varB="bar" '$1~var{f=1} f{print $1} $1~varB{f=0}' file
foo
3
4
bar
foo
6
bar
but without sample input and expected output it's just a guess and this would not address the SHELL behavior you are seeing wrt use of > vs >>.
Here's what happened. I used an array to input into my variables. I set the counter for what I thought was the total length of the array. When the final iteration of the array was reached, there was a null value returned to awk for the variable. This caused it to print EVERYTHING. Once I correctly had a counter with the correct number of array elements the printing oddity ended.
As far as the > vs >> goes, I don't know. It did stop, but I wasn't as careful in documenting it. I think what happened is that I used $1 in the print command to save time, and with each line it printed at the end it erased the whole file and left the last three identical matches. Something to ponder. Thanks Ed for the honest work. And no thank you to Robo responses.
I am writing a script in bash which takes a parameter and storing it;
threshold = $1
I then have sample data that looks something like:
5 blargh
6 tree
2 dog
1 fox
9 fridge
I wish to print only the lines which have their number greater than the number which is entered as the parameter (threshold).
I am currently using:
awk '{print $1 > $threshold}' ./file
But nothing prints out, help would be appreciated.
You're close, but it needs to be more like this:
$ threshold=3
$ awk -v threshold="$threshold" '$1 > threshold' file
Creating a variable with -v avoids the ugliness of trying to expand shell variables within an awk script.
EDIT:
There are a few problems with the current code you've shown. The first is that your awk script is single quoted (good), which stops $threshold from expanding, and so the value is never inserted in your script. Second, your condition belongs outside the curly braces, which would make it:
$1 > threshold { print }
This works, but the `print is not necessary (it's the default action), which is why I shortened it to
$1 > threshold
I'm trying to write a shell script and plan to calculate a simple division using two variables inside the script. I couldn't get it to work. It's some kind of syntax error.
Here is part of my code, named test.sh
awk '{a+=$5} END {print a}' $variable1 > casenum
awk '{a+=$5} END {print a}' $variable2 > controlnum
score=$(echo "scale=4; $casenum/$controlnum" | bc)
printf "%s\t%s\t%.4f\n", $variable3 $variable4 $score
It's just the $score that doesn't work.
I tried to use either
sh test.sh
or
bash test.sh
but neither worked. The error message is:
(standard_in) 1: syntax error
Does anyone know how to make it work? Thanks so much!
You are outputting to files, not to vars. For this, you need var=$(command). Hence, this should make it:
casenum=$(awk '{a+=$5} END {print a}' $variable1)
controlnum=$(awk '{a+=$5} END {print a}' $variable2)
score=$(echo "scale=4; $casenum/$controlnum" | bc)
printf "%s\t%s\t%.4f\n", $variable3 $variable4 $score
Note $variable1 and $variable2 should be file names. Otherwise, indicate it.
First your $variable1 and $variable2 must expand to a name of an existing file; but that's not a syntax error, it's just a fact that makes your code wrong, unless you mean really to cope with files containing numbers and accumulating the sum of the fifth field into a file. Since casenum and controlnum are not assigned (in fact you write the awk result to a file, not into a variable), your score computation expands to
score=$(echo "scale=4; /" | bc)
which is wrong (Syntax error comes from this).
Then, the same problem with $variable3 and $variable4. Are they holding a value? Have you assigned them with something like
variable=...
? Otherwise they will expand as "". Fixing these (including assigning casenum and controlnum), will fix everything, since basically the only syntax error is when bc tries to interpret the command / without operands. (And the comma after the printf is not needed).
The way you assign the output of execution of a command to a variable is
var=$(command)
or
var=`command`
If I understand your commands properly, you could combine calculation of score with a single awk statement as follows
score=$(awk 'NR==FNR {a+=$5; next} {b+=$5} END {printf "%.4f", a/b}' $variable1 $variable2)
This is with assumption that $variable1 and $variable2 are valid file names
Refer to #fedorqui's solution if you want to stick to your approach of 2 awk and 1 bc.
I am using a shell script and within that I am using an awk script. I am passing parameters to awk from shell script by using -v option. At some point of time, when the argument size exceeds a certain limit, I was getting the 'Argument list too long error'. This was my previous question but I have found out the root cause for the same. Now my question is:
Variable to be passed from shell to awk using -v option = too large ⟶ Hence getting argument list too long error
My idea is to break the large variable into small chunks and store it in an array and then pass the array into awk instead of passing the single variable into awk.
My question is:
Is it possible to break the large variable into small array and then pass it back to awk. I know how to modify a variable of shell inside an awk script. But how can I modify the array of shell inside an awk script?
I read that -v option is not advisable and they suggested to pipe the variable values. So if that the case
echo variable | awk '{}'
So variables would be piped. But I have to pipe an array along with some other variables. Could you please help me?
CODE DESCRIPTION
addvariable=""
export variable
loop begins
eval $(awk -v tempvariable="$addvariable" '{tempvariable=tempvariable+"long string" variable=tempvariable(Here is where the shell variable(variable) is being modified )}')
In shell
addvariable=$variable (Taking the new value of shell variable and feeding back to awk in the next iteration)
loop ends
So the problem is now as the addvariable and variable keeps on increasing, I am get argument too long error .. So what I have to do is to split the tempvariable into small chunks and then store it in variable[1] variable[2] etc and then assign that to addvariable[1], addvariable[2] and the feed addvariable[1],[2] instead of feeding the entire addvariable as a whole.So my question is how to feed that as an array. and how to store the big data inside the awk into variable[1] variable[2]
CODE
addshellvariable=""
for i in {0..10}
{
zcat normalfile{i} > FILE A
zcat hugefile{i} > FILE
export shellvariable=""
getdate=grep "XXX" FILE B|sort|Uniq (getdate contains a list of id's)
eval $(awk -v getdata="$getdata" -v addshellvariable="$addshellvariable" BEGIN {tempvariable="";split(addshellvariable,tempshellvariableArray,"*");while(t <= length(tempshellvariable)) {awkarray[tempshellvariableArray[t]];} {for(id in ids) {awkarray[id];} END {for(id in awkarray) {tempvariable=tempvariable"*"id"*"awkarray[id]} **print "shellvariable"=tempvariable;**}} FILE A)
addshellvariable=$shellvariable;
}
So as you can see awk is being embedded inside the shell . everytime I need the awkarray content to be feedback into the awk again .. So that I will be able to get the updated ones and that is the reason I am getting the awk array content in the shell variable by printing that, again the shell variable is stored in an another shell variable "addshellvariable" and that is being given to the awk in the next iteration. But the problem is when the shellvariable size increases a certain point then I am getting an Argument too long error . Thus I wanted a solution in such a way that , instead of doing
print "shellvariable"=tempvariable; I can make it as print "shellvariable[1]"=A part of tempvariable; and so on ...
Your shell appears to have limited you. I suspect that your guess is correct, and this isn't an awk problem, it's the scripting language from which you're calling awk.
You can pre-load awk with variables loaded from a file. Check this out:
$ printf 'foo=2\nbar=3\nbaz=4\n' > vars
$ printf 'snarf\nblarg\nbaz\nsnurry\n' > text
$ awk 'NR==FNR{split($0,a,"=");vars[a[1]]=a[2];next} $1 in vars {print vars[$1]}' vars text
4
$
How does this work?
The first two printf lines give us our raw data. Run them without the redirect (or cat the resultant files) if they're not completely clear to you.
The awk script has two main sections. Awk scripts consist of repetitions of condition { commands }. In this case, we've got two of these sets.
The first set has a condition of NR==FNR. This evaluates as "true" if the current record number that awk is processing (NR) is the same as the current record number in the current file. Obviously, this only works for the FIRST file, because as of the first line in the second file, NR is 1 plus the line count of the first file.
Within this section, we split() the line according to its equals sign, and put the data into an array called vars.
The second set has a condition of $1 in vars, which evaluates to true if the first word of the current line exists as a subscript of the vars array. I include this only as an example of what you can do with vars, since I don't know what you're trying to achieve with these variables.
Does this address your problem? If not, we'll need to see some of your code to get an idea of how to fix it.
UPDATE per suggestion in comments, here's proof that it works for large variables:
First, we prepare our input data:
$ dd if=/dev/random of=out.rand count=128k bs=1k
131072+0 records in
131072+0 records out
134217728 bytes transferred in 3.265765 secs (41098404 bytes/sec)
$ b64encode -o out.b64 out.rand out.rand
$ ls -lh out.b64
-rw-r--r-- 1 ghoti wheel 172M Jul 17 01:08 out.b64
$ awk 'BEGIN{printf("foo=")} NR>1{printf("%s",$0)} END{print ""}' out.b64 > vars
$ ls -lh vars
-rw-r--r-- 1 ghoti wheel 170M Jul 17 01:10 vars
$ wc -l vars
1 vars
$ cut -c1-30 vars
foo=orq0UgQJyUAcwJV0SenJrSHu3j
Okay, we've got a ~170MB variable on a single line. Let's suck it into awk.
$ awk 'NR==FNR{split($0,a,"=");vars[a[1]]=a[2];next} END{print length(vars["foo"]);print "foo=" substr(vars["foo"],0,26);}' out.var bar
178956971
foo=orq0UgQJyUAcwJV0SenJrSHu3j
We can see the size of the variable, and the first 26 characters match what we saw from shell. Yup, it works.
I have created simple script:
#!/bin/sh
column=${1:-1}
awk '{colawk='$column'+1; print colawk}'
But when I run:
ls -la | ./Column.sh 4
I receive output:
5
5
5
5
But I have expected receive 5th column. Why this error?
I believe this will do what you've attempted in your example:
#!/bin/sh
let "column=${1:-1} + 1"
awk "{print \$$column}"
However, I don't see why you're adding one to the column index? You'll then not be able to intuitively access the first column.
I'd to it this way instead:
#!/bin/sh
let "column=${1:-1}"
awk "{print \$$column}"
The argument to ./Column.sh will be the column number you want, 0 will give you all columns, while a call without arguments will default the column index to 1.
I know bash. I would like make arithmetic with AWK
In that case, how about:
#!/bin/sh
column=${1:-1}
awk 'BEGIN{colawk='$column'+1} {print $colawk}'
Or, simply:
#!/bin/sh
awk 'BEGIN{colawk='${1:-1}'+1} {print $colawk}'
Two things I changed in your script:
put the arithmetic in a BEGIN{} block since it only needs to be done once and not repeated for every input line.
"print $colawk" instead of "print colawk" so we're printing the column indexed by colawk instead of its value.