cat TEXT | awk -v var=$i -v varB=$j '$1~var , $1~varB {print $1}' > PROBLEM HERE
I am passing two variables from an array to parse a very large text file by range. And it works, kind of.
if I use ">" the output to the file will ONLY be the last three lines as verified by cat and a text editor.
if I use ">>" the output to the file will include one complete read of TEXT and then it will divide the second read into the ranges I want.
if I let the output go through to the shell I get the same problem as above.
Question:
It appears awk is reading every line and printing it. Then it goes back and selects the ranges from the TEXT file. It does not do this if I use constants in the range pattern search.
I undestand awk must read all lines to find the ranges I request.
why is it printing the entire document?
How can I get it to ONLY print the ranges selected?
This is the last hurdle in a big project and I am beating my head against the table.
Thanks!
give this a try, you didn't assign varB in right way:
yours: awk -v var="$i" -varB="$j" ...
mine : awk -v var="$i" -v varB="$j" ...
^^
Aside from the typo, you can't use variables in //, instead you have to specify with regular ~ match. Also quote your shell variables (here is not needed obviously, but to set an example). For example
seq 1 10 | awk -v b="3" -v e="5" '$0 ~ b, $0 ~ e'
should print 3..5 as expected
It sounds like this is what you want:
awk -v var="foo" -v varB="bar" '$1~var{f=1} f{print $1} $1~varB{f=0}' file
e.g.
$ cat file
1
2
foo
3
4
bar
5
foo
6
bar
7
$ awk -v var="foo" -v varB="bar" '$1~var{f=1} f{print $1} $1~varB{f=0}' file
foo
3
4
bar
foo
6
bar
but without sample input and expected output it's just a guess and this would not address the SHELL behavior you are seeing wrt use of > vs >>.
Here's what happened. I used an array to input into my variables. I set the counter for what I thought was the total length of the array. When the final iteration of the array was reached, there was a null value returned to awk for the variable. This caused it to print EVERYTHING. Once I correctly had a counter with the correct number of array elements the printing oddity ended.
As far as the > vs >> goes, I don't know. It did stop, but I wasn't as careful in documenting it. I think what happened is that I used $1 in the print command to save time, and with each line it printed at the end it erased the whole file and left the last three identical matches. Something to ponder. Thanks Ed for the honest work. And no thank you to Robo responses.
Related
System: Linux. Bash 4.
I have the following file, which will be read into a script as a variable:
/path/sample_A.bam A 1
/path/sample_B.bam B 1
/path/sample_C1.bam C 1
/path/sample_C2.bam C 2
I want to append "_string" at the end of the filename of the first column, but before the extension (.bam). It's a bit trickier because of containing the path at the beginning of the name.
Desired output:
/path/sample_A_string.bam A 1
/path/sample_B_string.bam B 1
/path/sample_C1_string.bam C 1
/path/sample_C2_string.bam C 2
My attempt:
I did the following script (I ran: bash script.sh):
List=${1};
awk -F'\t' -vOFS='\t' '{ $1 = "${1%.bam}" "_string.bam" }1' < ${List} ;
And its output was:
${1%.bam}_string.bam
${1%.bam}_string.bam
${1%.bam}_string.bam
${1%.bam}_string.bam
Problem:
I followed the idea of using awk for this substitution as in this thread https://unix.stackexchange.com/questions/148114/how-to-add-words-to-an-existing-column , but the parameter expansion of ${1%.bam} it's clearly not being recognised by AWK as I intend. Does someone know the correct syntax for that part of code? That part was meant to mean "all the first entry of the first column, except the last part of .bam". I used ${1%.bam} because it works in Bash, but AWK it's another language and probably this differs. Thank you!
Note that the paramter expansion you applied on $1 won't apply inside awk as the entire command
body of the awk command is passed in '..' which sends content literally without applying any
shell parsing. Hence the string "${1%.bam}" is passed as-is to the first column.
You can do this completely in Awk
awk -F'\t' 'BEGIN { OFS = FS }{ n=split($1, arr, "."); $1 = arr[1]"_string."arr[2] }1' file
The code basically splits the content of $1 with delimiter . into an array arr in the context of Awk. So the part of the string upto the first . is stored in arr[1] and the subsequent split fields are stored in the next array indices. We re-construct the filename of your choice by concatenating the array entries with the _string in the filename part without extension.
If I understood your requirement correctly, could you please try following.
val="_string"
awk -v value="$val" '{sub(".bam",value"&")} 1' Input_file
Brief explanation: -v value means passing shell variable named val value to awk variable variable here. Then using sub function of awk to substitute string .bam with string value along with .bam value which is denoted by & too. Then mentioning 1 means print edited/non-edtied line.
Why OP's attempt didn't work: Dear, OP. in awk we can't pass variables of shell directly without mentioning them in awk language. So what you are trying will NOT take it as an awk variable rather than it will take it as a string and printing it as it is. I have mentioned in my explanation above how to define shell variables in awk too.
NOTE: In case you have multiple occurences of .bam then please change sub to gsub in above code. Also in case your Input_file is TAB delmited then use awk -F'\t' in above code.
sed -i 's/\.bam/_string\.bam/g' myfile.txt
It's a single line with sed. Just replace the .bam with _string.bam
You can try this way with awk :
awk -v a='_string' 'BEGIN{FS=OFS="."}{$1=$1 a}1' infile
I am writing a script in bash which takes a parameter and storing it;
threshold = $1
I then have sample data that looks something like:
5 blargh
6 tree
2 dog
1 fox
9 fridge
I wish to print only the lines which have their number greater than the number which is entered as the parameter (threshold).
I am currently using:
awk '{print $1 > $threshold}' ./file
But nothing prints out, help would be appreciated.
You're close, but it needs to be more like this:
$ threshold=3
$ awk -v threshold="$threshold" '$1 > threshold' file
Creating a variable with -v avoids the ugliness of trying to expand shell variables within an awk script.
EDIT:
There are a few problems with the current code you've shown. The first is that your awk script is single quoted (good), which stops $threshold from expanding, and so the value is never inserted in your script. Second, your condition belongs outside the curly braces, which would make it:
$1 > threshold { print }
This works, but the `print is not necessary (it's the default action), which is why I shortened it to
$1 > threshold
I'm pulling in loads of data from a network and filtering for foo and bar, e.g.
for i in example.com example.org example.net
do
echo "Data from $i"
curl $i/data.csv | grep --after-context=3 "foo|bar"
done
Every time foo appears, I need to see the next few lines (grep --after-context=3), but when bar appears, I only need that single line.
Is it possible to make it work in a single grep, sed, awk (or other standard unix) command?
One way:
curl .... | awk '/foo/{x=NR+3}(NR<=x) || /bar/'
When foo is encountered, x is set to current line number + 3, and hence the condition (NR+x) makes the line "foo" and the next 3 lines get printed. /bar/ makes the line containing the bar printed.
awk 'BEGIN {np=0} /bar/ {print; next} /foo/ {np=1;ln=RN;print;next} ln!=0 && RN>(ln+3) {np=0;ln=0} np==1 {print}' INPUTFILE
Instead of the grep, you might use the above. What it does:
in BEGIN sets up the non printing variable.
/bar/ {print} if you can't figure this out, well... (the next is for skipping every other rules and move to the next record).
/foo/ {np=1;ln=RN;print} prints foo lines, saves the line number, and sets print later lines
if the actual row number is greater than the saved line number plus 3, sets printing to off
if we need to print (np>0), then print.
This might work for you (GNU sed);
sed -n '/foo/,+3{p;b};/bar/p' file
I am using a shell script and within that I am using an awk script. I am passing parameters to awk from shell script by using -v option. At some point of time, when the argument size exceeds a certain limit, I was getting the 'Argument list too long error'. This was my previous question but I have found out the root cause for the same. Now my question is:
Variable to be passed from shell to awk using -v option = too large ⟶ Hence getting argument list too long error
My idea is to break the large variable into small chunks and store it in an array and then pass the array into awk instead of passing the single variable into awk.
My question is:
Is it possible to break the large variable into small array and then pass it back to awk. I know how to modify a variable of shell inside an awk script. But how can I modify the array of shell inside an awk script?
I read that -v option is not advisable and they suggested to pipe the variable values. So if that the case
echo variable | awk '{}'
So variables would be piped. But I have to pipe an array along with some other variables. Could you please help me?
CODE DESCRIPTION
addvariable=""
export variable
loop begins
eval $(awk -v tempvariable="$addvariable" '{tempvariable=tempvariable+"long string" variable=tempvariable(Here is where the shell variable(variable) is being modified )}')
In shell
addvariable=$variable (Taking the new value of shell variable and feeding back to awk in the next iteration)
loop ends
So the problem is now as the addvariable and variable keeps on increasing, I am get argument too long error .. So what I have to do is to split the tempvariable into small chunks and then store it in variable[1] variable[2] etc and then assign that to addvariable[1], addvariable[2] and the feed addvariable[1],[2] instead of feeding the entire addvariable as a whole.So my question is how to feed that as an array. and how to store the big data inside the awk into variable[1] variable[2]
CODE
addshellvariable=""
for i in {0..10}
{
zcat normalfile{i} > FILE A
zcat hugefile{i} > FILE
export shellvariable=""
getdate=grep "XXX" FILE B|sort|Uniq (getdate contains a list of id's)
eval $(awk -v getdata="$getdata" -v addshellvariable="$addshellvariable" BEGIN {tempvariable="";split(addshellvariable,tempshellvariableArray,"*");while(t <= length(tempshellvariable)) {awkarray[tempshellvariableArray[t]];} {for(id in ids) {awkarray[id];} END {for(id in awkarray) {tempvariable=tempvariable"*"id"*"awkarray[id]} **print "shellvariable"=tempvariable;**}} FILE A)
addshellvariable=$shellvariable;
}
So as you can see awk is being embedded inside the shell . everytime I need the awkarray content to be feedback into the awk again .. So that I will be able to get the updated ones and that is the reason I am getting the awk array content in the shell variable by printing that, again the shell variable is stored in an another shell variable "addshellvariable" and that is being given to the awk in the next iteration. But the problem is when the shellvariable size increases a certain point then I am getting an Argument too long error . Thus I wanted a solution in such a way that , instead of doing
print "shellvariable"=tempvariable; I can make it as print "shellvariable[1]"=A part of tempvariable; and so on ...
Your shell appears to have limited you. I suspect that your guess is correct, and this isn't an awk problem, it's the scripting language from which you're calling awk.
You can pre-load awk with variables loaded from a file. Check this out:
$ printf 'foo=2\nbar=3\nbaz=4\n' > vars
$ printf 'snarf\nblarg\nbaz\nsnurry\n' > text
$ awk 'NR==FNR{split($0,a,"=");vars[a[1]]=a[2];next} $1 in vars {print vars[$1]}' vars text
4
$
How does this work?
The first two printf lines give us our raw data. Run them without the redirect (or cat the resultant files) if they're not completely clear to you.
The awk script has two main sections. Awk scripts consist of repetitions of condition { commands }. In this case, we've got two of these sets.
The first set has a condition of NR==FNR. This evaluates as "true" if the current record number that awk is processing (NR) is the same as the current record number in the current file. Obviously, this only works for the FIRST file, because as of the first line in the second file, NR is 1 plus the line count of the first file.
Within this section, we split() the line according to its equals sign, and put the data into an array called vars.
The second set has a condition of $1 in vars, which evaluates to true if the first word of the current line exists as a subscript of the vars array. I include this only as an example of what you can do with vars, since I don't know what you're trying to achieve with these variables.
Does this address your problem? If not, we'll need to see some of your code to get an idea of how to fix it.
UPDATE per suggestion in comments, here's proof that it works for large variables:
First, we prepare our input data:
$ dd if=/dev/random of=out.rand count=128k bs=1k
131072+0 records in
131072+0 records out
134217728 bytes transferred in 3.265765 secs (41098404 bytes/sec)
$ b64encode -o out.b64 out.rand out.rand
$ ls -lh out.b64
-rw-r--r-- 1 ghoti wheel 172M Jul 17 01:08 out.b64
$ awk 'BEGIN{printf("foo=")} NR>1{printf("%s",$0)} END{print ""}' out.b64 > vars
$ ls -lh vars
-rw-r--r-- 1 ghoti wheel 170M Jul 17 01:10 vars
$ wc -l vars
1 vars
$ cut -c1-30 vars
foo=orq0UgQJyUAcwJV0SenJrSHu3j
Okay, we've got a ~170MB variable on a single line. Let's suck it into awk.
$ awk 'NR==FNR{split($0,a,"=");vars[a[1]]=a[2];next} END{print length(vars["foo"]);print "foo=" substr(vars["foo"],0,26);}' out.var bar
178956971
foo=orq0UgQJyUAcwJV0SenJrSHu3j
We can see the size of the variable, and the first 26 characters match what we saw from shell. Yup, it works.
I have created simple script:
#!/bin/sh
column=${1:-1}
awk '{colawk='$column'+1; print colawk}'
But when I run:
ls -la | ./Column.sh 4
I receive output:
5
5
5
5
But I have expected receive 5th column. Why this error?
I believe this will do what you've attempted in your example:
#!/bin/sh
let "column=${1:-1} + 1"
awk "{print \$$column}"
However, I don't see why you're adding one to the column index? You'll then not be able to intuitively access the first column.
I'd to it this way instead:
#!/bin/sh
let "column=${1:-1}"
awk "{print \$$column}"
The argument to ./Column.sh will be the column number you want, 0 will give you all columns, while a call without arguments will default the column index to 1.
I know bash. I would like make arithmetic with AWK
In that case, how about:
#!/bin/sh
column=${1:-1}
awk 'BEGIN{colawk='$column'+1} {print $colawk}'
Or, simply:
#!/bin/sh
awk 'BEGIN{colawk='${1:-1}'+1} {print $colawk}'
Two things I changed in your script:
put the arithmetic in a BEGIN{} block since it only needs to be done once and not repeated for every input line.
"print $colawk" instead of "print colawk" so we're printing the column indexed by colawk instead of its value.