I am writing a script in bash which takes a parameter and storing it;
threshold = $1
I then have sample data that looks something like:
5 blargh
6 tree
2 dog
1 fox
9 fridge
I wish to print only the lines which have their number greater than the number which is entered as the parameter (threshold).
I am currently using:
awk '{print $1 > $threshold}' ./file
But nothing prints out, help would be appreciated.
You're close, but it needs to be more like this:
$ threshold=3
$ awk -v threshold="$threshold" '$1 > threshold' file
Creating a variable with -v avoids the ugliness of trying to expand shell variables within an awk script.
EDIT:
There are a few problems with the current code you've shown. The first is that your awk script is single quoted (good), which stops $threshold from expanding, and so the value is never inserted in your script. Second, your condition belongs outside the curly braces, which would make it:
$1 > threshold { print }
This works, but the `print is not necessary (it's the default action), which is why I shortened it to
$1 > threshold
Related
I have some output files (5000 files) of .log which are the results of QM computations. Inside each file there are two special lines indicate the number of electrons and orbitals, like this below as an example (with exact spaces as in output files):
Number of electrons = 9
Number of orbitals = 13
I thought about a script (bash or Fortran), as a solution to this problem, which grep these two lines (at same time) and get the corresponding integer values (9 and 13, for instance), compare them and finds the difference between two values, and finally, list them in a new text file with the corresponding filenames.
I would really appreciate any help given.
Am posting an attempt in GNU Awk, and have tested it in that only.
#!/bin/bash
for file in *.log
do
awk -F'=[[:blank:]]*' '/Number of/{printf "%s%s",$2,(NR%2?" ":RS)}' "$file" | awk 'function abs(v) {return v < 0 ? -v : v} {print abs($1-$2)}' >> output_"$file"
done
The reason I split the AWK logic to two was to reduce the complexity in doing it in single huge command. The first part is for extracting the numbers from your log file in a columnar format and second for getting their absolute value.
I will break-down the AWK logic:-
-F'=[[:blank:]]*' is a mult0 character delimiter logic including = and one or more instances of [[:blank:]] whitespace characters.
'/Number of/{printf "%s%s",$2,(NR%2?" ":RS)}' searches for lines starting with Number of and prints it in a columnar fashion, i.e. as 9 13 from your sample file.
The second part is self-explanatory. I have written a function to get the absolute value from the two returned values and print it.
Each output is saved in a file named output_, for you to process it further.
Run the script from your command line as bash script.sh, where script.sh is the name containing the above lines.
Update:-
In case if you are interested in negative values too i.e. without the absolute function, change the awk statement to
awk -F'=[[:blank:]]*' '/Number of/{printf "%s%s",$2,(NR%2?" ":RS)}' "$file" | awk '{print ($1-$2)}' >> output_"$file"
Bad way to do it (but it will work)-
while read file
do
first=$(awk -F= '/^Number/ {print $2}' "$file" | head -1)
second=$(awk -F= '/^Number/ {print $2}' "$file" | tail -1)
if [ "$first" -gt "$second" ]
then
echo $(("$first" - "$second"))
else
echo $(("$second" - "$first"))
fi > "$file"_answer ;
done < list_of_files
This method picks up the values (in the awk one liner and compares them.
It then subtracts them to give you one value which it saves in the file called "$file"_answer. i.e. the initial file name with '_answer' as a suffix to the name.
You may need to tweak this code to fit your purposes exactly.
cat TEXT | awk -v var=$i -v varB=$j '$1~var , $1~varB {print $1}' > PROBLEM HERE
I am passing two variables from an array to parse a very large text file by range. And it works, kind of.
if I use ">" the output to the file will ONLY be the last three lines as verified by cat and a text editor.
if I use ">>" the output to the file will include one complete read of TEXT and then it will divide the second read into the ranges I want.
if I let the output go through to the shell I get the same problem as above.
Question:
It appears awk is reading every line and printing it. Then it goes back and selects the ranges from the TEXT file. It does not do this if I use constants in the range pattern search.
I undestand awk must read all lines to find the ranges I request.
why is it printing the entire document?
How can I get it to ONLY print the ranges selected?
This is the last hurdle in a big project and I am beating my head against the table.
Thanks!
give this a try, you didn't assign varB in right way:
yours: awk -v var="$i" -varB="$j" ...
mine : awk -v var="$i" -v varB="$j" ...
^^
Aside from the typo, you can't use variables in //, instead you have to specify with regular ~ match. Also quote your shell variables (here is not needed obviously, but to set an example). For example
seq 1 10 | awk -v b="3" -v e="5" '$0 ~ b, $0 ~ e'
should print 3..5 as expected
It sounds like this is what you want:
awk -v var="foo" -v varB="bar" '$1~var{f=1} f{print $1} $1~varB{f=0}' file
e.g.
$ cat file
1
2
foo
3
4
bar
5
foo
6
bar
7
$ awk -v var="foo" -v varB="bar" '$1~var{f=1} f{print $1} $1~varB{f=0}' file
foo
3
4
bar
foo
6
bar
but without sample input and expected output it's just a guess and this would not address the SHELL behavior you are seeing wrt use of > vs >>.
Here's what happened. I used an array to input into my variables. I set the counter for what I thought was the total length of the array. When the final iteration of the array was reached, there was a null value returned to awk for the variable. This caused it to print EVERYTHING. Once I correctly had a counter with the correct number of array elements the printing oddity ended.
As far as the > vs >> goes, I don't know. It did stop, but I wasn't as careful in documenting it. I think what happened is that I used $1 in the print command to save time, and with each line it printed at the end it erased the whole file and left the last three identical matches. Something to ponder. Thanks Ed for the honest work. And no thank you to Robo responses.
I am extracting user ratings after making some calculations. If calculated rating exceeds 5.0 (say 5.001), I want to clip it to 5 and if it falls below 1 (say 0.001), I want to limit it to 1.
I will be grateful if there can be a bash script that makes this clipping. Or maybe I can use a python script from command line which takes dollar variable from bash.
You can use awk:
awk '$0<1{$0=1}$0>5{$0=5}1' <<<"$var"
If the variable is less than 1, change it to 1. If it is greater than 5, change it to 5. 1 at the end is always true, so awk performs the default action, which is to print the record.
Some additional explanation:
<<< is an alternative way of writing echo "$var" | awk ..., supported by bash
$0 refers to the current "record", which in this case is the variable that has been echoed to bash
awk works like condition { action }, so each block is only run if the condition before it is true
if no {action} part is specified, the default action is {print}, so that's what the 1 at the end does.
For example:
$ var=0.001
$ awk '$0<1{$0=1}$0>5{$0=5}1' <<<"$var"
1
$ var=5.001
$ awk '$0<1{$0=1}$0>5{$0=5}1' <<<"$var"
5
To overwrite the variable, you can use command substitution:
var=$(awk '$0<1{$0=1}$0>5{$0=5}1' <<<"$var")
If you like confusing yourself, you can do the same thing using two ternary operators instead:
awk '{$0=$0<1?1:$0>5?5:$0}1' <<<"$var"
I am using a shell script and within that I am using an awk script. I am passing parameters to awk from shell script by using -v option. At some point of time, when the argument size exceeds a certain limit, I was getting the 'Argument list too long error'. This was my previous question but I have found out the root cause for the same. Now my question is:
Variable to be passed from shell to awk using -v option = too large ⟶ Hence getting argument list too long error
My idea is to break the large variable into small chunks and store it in an array and then pass the array into awk instead of passing the single variable into awk.
My question is:
Is it possible to break the large variable into small array and then pass it back to awk. I know how to modify a variable of shell inside an awk script. But how can I modify the array of shell inside an awk script?
I read that -v option is not advisable and they suggested to pipe the variable values. So if that the case
echo variable | awk '{}'
So variables would be piped. But I have to pipe an array along with some other variables. Could you please help me?
CODE DESCRIPTION
addvariable=""
export variable
loop begins
eval $(awk -v tempvariable="$addvariable" '{tempvariable=tempvariable+"long string" variable=tempvariable(Here is where the shell variable(variable) is being modified )}')
In shell
addvariable=$variable (Taking the new value of shell variable and feeding back to awk in the next iteration)
loop ends
So the problem is now as the addvariable and variable keeps on increasing, I am get argument too long error .. So what I have to do is to split the tempvariable into small chunks and then store it in variable[1] variable[2] etc and then assign that to addvariable[1], addvariable[2] and the feed addvariable[1],[2] instead of feeding the entire addvariable as a whole.So my question is how to feed that as an array. and how to store the big data inside the awk into variable[1] variable[2]
CODE
addshellvariable=""
for i in {0..10}
{
zcat normalfile{i} > FILE A
zcat hugefile{i} > FILE
export shellvariable=""
getdate=grep "XXX" FILE B|sort|Uniq (getdate contains a list of id's)
eval $(awk -v getdata="$getdata" -v addshellvariable="$addshellvariable" BEGIN {tempvariable="";split(addshellvariable,tempshellvariableArray,"*");while(t <= length(tempshellvariable)) {awkarray[tempshellvariableArray[t]];} {for(id in ids) {awkarray[id];} END {for(id in awkarray) {tempvariable=tempvariable"*"id"*"awkarray[id]} **print "shellvariable"=tempvariable;**}} FILE A)
addshellvariable=$shellvariable;
}
So as you can see awk is being embedded inside the shell . everytime I need the awkarray content to be feedback into the awk again .. So that I will be able to get the updated ones and that is the reason I am getting the awk array content in the shell variable by printing that, again the shell variable is stored in an another shell variable "addshellvariable" and that is being given to the awk in the next iteration. But the problem is when the shellvariable size increases a certain point then I am getting an Argument too long error . Thus I wanted a solution in such a way that , instead of doing
print "shellvariable"=tempvariable; I can make it as print "shellvariable[1]"=A part of tempvariable; and so on ...
Your shell appears to have limited you. I suspect that your guess is correct, and this isn't an awk problem, it's the scripting language from which you're calling awk.
You can pre-load awk with variables loaded from a file. Check this out:
$ printf 'foo=2\nbar=3\nbaz=4\n' > vars
$ printf 'snarf\nblarg\nbaz\nsnurry\n' > text
$ awk 'NR==FNR{split($0,a,"=");vars[a[1]]=a[2];next} $1 in vars {print vars[$1]}' vars text
4
$
How does this work?
The first two printf lines give us our raw data. Run them without the redirect (or cat the resultant files) if they're not completely clear to you.
The awk script has two main sections. Awk scripts consist of repetitions of condition { commands }. In this case, we've got two of these sets.
The first set has a condition of NR==FNR. This evaluates as "true" if the current record number that awk is processing (NR) is the same as the current record number in the current file. Obviously, this only works for the FIRST file, because as of the first line in the second file, NR is 1 plus the line count of the first file.
Within this section, we split() the line according to its equals sign, and put the data into an array called vars.
The second set has a condition of $1 in vars, which evaluates to true if the first word of the current line exists as a subscript of the vars array. I include this only as an example of what you can do with vars, since I don't know what you're trying to achieve with these variables.
Does this address your problem? If not, we'll need to see some of your code to get an idea of how to fix it.
UPDATE per suggestion in comments, here's proof that it works for large variables:
First, we prepare our input data:
$ dd if=/dev/random of=out.rand count=128k bs=1k
131072+0 records in
131072+0 records out
134217728 bytes transferred in 3.265765 secs (41098404 bytes/sec)
$ b64encode -o out.b64 out.rand out.rand
$ ls -lh out.b64
-rw-r--r-- 1 ghoti wheel 172M Jul 17 01:08 out.b64
$ awk 'BEGIN{printf("foo=")} NR>1{printf("%s",$0)} END{print ""}' out.b64 > vars
$ ls -lh vars
-rw-r--r-- 1 ghoti wheel 170M Jul 17 01:10 vars
$ wc -l vars
1 vars
$ cut -c1-30 vars
foo=orq0UgQJyUAcwJV0SenJrSHu3j
Okay, we've got a ~170MB variable on a single line. Let's suck it into awk.
$ awk 'NR==FNR{split($0,a,"=");vars[a[1]]=a[2];next} END{print length(vars["foo"]);print "foo=" substr(vars["foo"],0,26);}' out.var bar
178956971
foo=orq0UgQJyUAcwJV0SenJrSHu3j
We can see the size of the variable, and the first 26 characters match what we saw from shell. Yup, it works.
I have created simple script:
#!/bin/sh
column=${1:-1}
awk '{colawk='$column'+1; print colawk}'
But when I run:
ls -la | ./Column.sh 4
I receive output:
5
5
5
5
But I have expected receive 5th column. Why this error?
I believe this will do what you've attempted in your example:
#!/bin/sh
let "column=${1:-1} + 1"
awk "{print \$$column}"
However, I don't see why you're adding one to the column index? You'll then not be able to intuitively access the first column.
I'd to it this way instead:
#!/bin/sh
let "column=${1:-1}"
awk "{print \$$column}"
The argument to ./Column.sh will be the column number you want, 0 will give you all columns, while a call without arguments will default the column index to 1.
I know bash. I would like make arithmetic with AWK
In that case, how about:
#!/bin/sh
column=${1:-1}
awk 'BEGIN{colawk='$column'+1} {print $colawk}'
Or, simply:
#!/bin/sh
awk 'BEGIN{colawk='${1:-1}'+1} {print $colawk}'
Two things I changed in your script:
put the arithmetic in a BEGIN{} block since it only needs to be done once and not repeated for every input line.
"print $colawk" instead of "print colawk" so we're printing the column indexed by colawk instead of its value.