Remove multiple lines and following based on string in bash - bash

I have a (fasta) file input.fa that looks like this
>coucou
GAGAGATAGTATAGATATATAGGATATATA
>hello_world
GATATATTCTCTCTGAFAGACGACGACFGACTACTACGAC
>ziva_wesh
HAHTAHTAHTAHCGAGAGACAGCAGCAGCACTTACTACATCHBACAHCAHCAHA
I would like to get rid of both
>coucou
GAGAGATAGTATAGATATATAGGATATATA
and
>ziva_wesh
HAHTAHTAHTAHCGAGAGACAGCAGCAGCACTTACTACATCHBACAHCAHCAHA
What I am doing is (based on this solution by #Hai Vu)
$awk '/hello/{getline;next} 1' input.fa | awk '/coucou/{getline;next} 1'
>ziva_wesh
HAHTAHTAHTAHCGAGAGACAGCAGCAGCACTTACTACATCHBACAHCAHCAHA
Is there a way of doing this (using awk or sed or perl script) without "piping" the first awk result into a second awk command? (sthg like /hello&coucou/{getline;next} 1' input.fa)
Thanks for your answer!


One simple way:
$ awk '/hello/{getline;next} /coucou/{getline;next} 1' input.fa
>ziva_wesh
HAHTAHTAHTAHCGAGAGACAGCAGCAGCACTTACTACATCHBACAHCAHCAHA
Or if you prefer:
$ awk '/(hello)|(coucou)/{getline;next} 1' input.fa
>ziva_wesh
HAHTAHTAHTAHCGAGAGACAGCAGCAGCACTTACTACATCHBACAHCAHCAHA


A simple sed command can also handle this:
sed -nr '/>(hello|coucou)/{N;d};p' file
>ziva_wesh
HAHTAHTAHTAHCGAGAGACAGCAGCAGCACTTACTACATCHBACAHCAHCAHA


This might work for you (GNU sed):
sed -r '/>(coucou|ziva_wesh)/,+1d' file
This deletes the ranges of 2 lines (the match of the line containing >coucou or >ziva_wesh and the following line).

Related

How to convert multiple parameters URLs into single parameter URLs in bash

$ cat urls.txt
http://example.com/test/test/test?apple=&bat=&cat=&dog=
https://test.com/test/test/test?aa=&bb=&cc=
http://target.com/test/test?hmm=
I want output like below 👇🏻 , how can i do that in bash ( single line command )
$ cat urls.txt
http://example.com/test/test/test?apple=
http://example.com/test/test/test?bat=
http://example.com/test/test/test?cat=
http://example.com/test/test/test?dog=
https://test.com/test/test/test?aa=
https://test.com/test/test/test?bb=
https://test.com/test/test/test?cc=
http://target.com/test/test?hmm=

With GNU awk:
$ awk -F'?|=&|=' '{for(i=2;i<NF;i++) print $1 "?" $i "="}' urls.txt
http://example.com/test/test/test?apple=
http://example.com/test/test/test?bat=
http://example.com/test/test/test?cat=
http://example.com/test/test/test?dog=
https://test.com/test/test/test?aa=
https://test.com/test/test/test?bb=
https://test.com/test/test/test?cc=
http://target.com/test/test?hmm=

I try use sed but it is complex. if use perl like this:
perl -pe 'if(/(.*\?)/){$url=$1;s#&#\n$url#g;}' url.txt
it works well.

With GNU awk using gensub():
awk '{print gensub(/^(https?:)(.*)(\?[[:alpha:]]+=)(.*)/,"\\1\\2\\3","g")}' file
http://example.com/test/test/test?apple=
https://test.com/test/test/test?aa=
http://target.com/test/test?hmm=
gensub() for specifying components of the regexp in the replacement text, using parentheses in the regexp to mark the components (four here). We print only 3 of them: "\\1\\2\\3" .

This might work for you (GNU sed):
sed -E 's/(([^?]+\?)[^=]+=)&/\1\n\2/;P;D' file
Replace each & by a newline and the substring before the first parameter, print/delete the first line and repeat.

Using grep to pull a series of random numbers from a known line

I have a simple scalar file producing strings like...
bpred_2lev.ras_rate.PP 0.9413 # RAS prediction rate (i.e., RAS hits/used RAS)
Once I use grep to find this line in the output.txt, is there a way I can directly grab the "0.9413" portion? I am attempting to make a cvs file and just need whatever value is generated.
Thanks in advance.

There are several ways to combine finding and extracting into a single command:
awk (POSIX-compliant)
awk '$1 == "bpred_2lev.ras_rate.PP" { print $2 }' file
sed (GNU sed or BSD/OSX sed)
sed -En 's/^bpred_2lev\.ras_rate\.PP +([^ ]+).*$/\1/p' file
GNU grep
grep -Po '^bpred_2lev\.ras_rate\.PP +\K[^ ]+' file

You can use awk like this:
grep <your_search_criteria> output.txt | awk '{ print $2 }'

Extract string between two patterns (inclusive) while conserving the format

I have a file in the following format
cat test.txt
id1,PPLLTOMaaaaaaaaaaaJACK
id2,PPLRTOMbbbbbbbbbbbJACK
id3,PPLRTOMcccccccccccJACK
I am trying to identify and print the string between TOM and JACK including these two strings, while maintaining the first column FS=,
Desired output:
id1,TOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACK
id3,TOMcccccccccccJACK
So far I have tried gsub:
awk -F"," 'gsub(/.*TOM|JACK.*/,"",$2) && !_[$0]++' test.txt > out.txt
and have the following output
id1 aaaaaaaaaaa
id2 bbbbbbbbbbb
id3 ccccccccccc
As you can see I am getting close but not able to include TOM and JACK patterns in my output. Plus I am also losing the original FS. What am I doing wrong?
Any help will be appreciated.

You are changing a field ($2) which causes awk to reconstruct the record using the value of OFS as the field separator and so in this case changing the commas to spaces.
Never use _ as a variable name - using a name with no meaning is just slightly better than using a name with the wrong meaning, just pick a name that means something which, in this case is seen but idk what you are trying to do when using that in this context.
gsub() and sub() do not support capture groups so you either need to use match()+substr():
$ awk 'BEGIN{FS=OFS=","} match($2,/TOM.*JACK/){$2=substr($2,RSTART,RLENGTH)} 1' file
id1,TOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACK
id3,TOMcccccccccccJACK
or use GNU awk for the 3rd arg to match()
$ gawk 'BEGIN{FS=OFS=","} match($2,/TOM.*JACK/,a){$2=a[0]} 1' file
id1,TOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACK
id3,TOMcccccccccccJACK
or for gensub():
$ gawk 'BEGIN{FS=OFS=","} {$2=gensub(/.*(TOM.*JACK).*/,"\\1","",$2)} 1' file
id1,TOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACK
id3,TOMcccccccccccJACK
The main difference between the match() and gensub() solutions is how they would behave if TOM appeared twice on the line:
$ cat file
id1,PPLLfooTOMbarTOMaaaaaaaaaaaJACK
id2,PPLRTOMbbbbbbbbbbbJACKfooJACKbar
id3,PPLRfooTOMbarTOMcccccccccccJACKfooJACKbar
$
$ awk 'BEGIN{FS=OFS=","} match($2,/TOM.*JACK/,a){$2=a[0]} 1' file
id1,TOMbarTOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACKfooJACK
id3,TOMbarTOMcccccccccccJACKfooJACK
$
$ awk 'BEGIN{FS=OFS=","} {$2=gensub(/.*(TOM.*JACK).*/,"\\1","",$2)} 1' file
id1,TOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACKfooJACK
id3,TOMcccccccccccJACKfooJACK
and just to show one way of stopping at the first instead of the last JACK on the line:
$ awk 'BEGIN{FS=OFS=","} match($2,/TOM.*JACK/,a){$2=gensub(/(JACK).*/,"\\1","",a[0])} 1' file
id1,TOMbarTOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACK
id3,TOMbarTOMcccccccccccJACK

Use capture groups to save the parts of the line you want to keep. Here's how to do it with sed
sed 's/^\([^,]*,\).*\(TOM.*JACK\).*/\1\2/' <test.txt > out.txt

Do you mean to do the following?
$ cat test.txt
id1,PPLLTOMaaaaaaaaaaaJACKABCD
id2,PPLRTOMbbbbbbbbbbbJACKDFCC
id3,PPLRTOMcccccccccccJACKSDER
$ cat test.txt | sed -e 's/,.*TOM/,TOM/g' | sed -e 's/JACK.*/JACK/g'
id1,TOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACK
id3,TOMcccccccccccJACK
$
This should work as long as the TOM and JACK do not repeat themselves.

sed 's/\(.*,\).*\(TOM.*JACK\).*/\1\2/' <oldfile >newfile
Output:
id1,TOMaaaaaaaaaaaJACK
id2,TOMbbbbbbbbbbbJACK
id3,TOMcccccccccccJACK

Replace everything between two character

All.
I am newbie to sed.
I want something like
Input:
ABC,DEF,GHI,JKL,MNO
Output:
ABC,,,,MNO
Means....
I want to remove all contents between two ','

This might work for you (GNU sed):
sed 's/[^,]*,/,/2g' file

you could set all fields between 1 and last to empty with awk:
awk -F, -v OFS="," '{for(i=2;i<NF;i++)$i=""}7'

insert a blank line between every two lines in a file using shell, sed or awk

I have a file with many lines. I want to insert a blank line between each two lines
for example
original file
xfdljflsad
fjdiaopqqq
dioapfdja;
I want to make it as:
xfdljflsad
fjdiaopqqq
dioapfdja;
how to achieve this?
I want to use shell script, awk or sed for this?
thanks!

With sed, use
sed G input-file
If pilcrow is correct and you do not want an additional newline at the end of the file,
then do:
sed '$!G' input-file
Another alternative is to use pr:
pr -dt input-file

awk '{print nl $0; nl="\n"}' file

My approach if I want to quickly regex a file.
vim file.txt
%s/\n/\n\n/g

Idiomatic awk:
awk 1 ORS='\n\n' file
Similar thing with perl:
perl -nE 'say' file
Append | head -n -1 if final newline is unwanted.

Resources