Hi my files look like:
>ID.1
GGAACACGACATCCTGCAGGGTTAAAAAAGAAAAAATCAGTAAAAGTACTGGA
>ID.2
GGAATACCACATCCCGCAGGGTTAAAAAAGAAAAAATCAGTAACAGTACTGGA
and I want to move the lines so that line 1 swaps with 3, and line 2 swaps with 4.
>ID.2
GGAATACCACATCCCGCAGGGTTAAAAAAGAAAAAATCAGTAACAGTACTGGA
>ID.1
GGAACACGACATCCTGCAGGGTTAAAAAAGAAAAAATCAGTAAAAGTACTGGA
I have thought about using cut so cut send the lines into other files, and then bring them all back in the desired order using paste, but is there a solution using awk/sed.
EDIT: The file always has 4 lines (2 fasta entrys), no more.
For such a simple case, as #Ed_Morton mentioned, you can just swap the even-sized slices with head and tail commands:
$ tail -2 test.txt; head -2 test.txt
>ID.2
GGAATACCACATCCCGCAGGGTTAAAAAAGAAAAAATCAGTAACAGTACTGGA
>ID.1
GGAACACGACATCCTGCAGGGTTAAAAAAGAAAAAATCAGTAAAAGTACTGGA
Generic solution with GNU tac to reverse contents:
$ tac -bs'>' ip.txt
>ID.2
GGAATACCACATCCCGCAGGGTTAAAAAAGAAAAAATCAGTAACAGTACTGGA
>ID.1
GGAACACGACATCCTGCAGGGTTAAAAAAGAAAAAATCAGTAAAAGTACTGGA
By default tac reverses line wise but you can customize the separator.
Here, I'm assuming > can be safely used as a unique separator (provided to the -s option). The -b option is used to put the separator before the content in the output.
Using ed (inplace editing):
# move 3rd to 4th lines to the top
printf '3,4m0\nwq\n' | ed -s ip.txt
# move the last two lines to the top
printf -- '-1,$m0\nwq\n' | ed -s ip.txt
Using sed:
sed '1h;2H;1,2d;4G'
Store the first line in the hold space;
Add the second line to the hold space;
Don't print the first two lines;
Before printing the fourth line, append the hold space to it (i.e. append the 1st and 2nd line).
GNU AWK manual has example of swapping two lines using getline as you know that
The file always has 4 lines (2 fasta entrys), no more.
then you might care only about case when number of lines is evenly divisble by 4 and use getline following way, let file.txt content be
>ID.1
GGAACACGACATCCTGCAGGGTTAAAAAAGAAAAAATCAGTAAAAGTACTGGA
>ID.2
GGAATACCACATCCCGCAGGGTTAAAAAAGAAAAAATCAGTAACAGTACTGGA
then
awk '{line1=$0;getline line2;getline line3;getline line4;printf "%s\n%s\n%s\n%s\n",line3,line4,line1,line2}' file.txt
gives output
>ID.2
GGAATACCACATCCCGCAGGGTTAAAAAAGAAAAAATCAGTAACAGTACTGGA
>ID.1
GGAACACGACATCCTGCAGGGTTAAAAAAGAAAAAATCAGTAAAAGTACTGGA
Explanation: store current line in variable $0, then next line as line2, yet next line as line3, yet next line as line4, use printf with 4 placeholders (%s) followed by newlines (\n), which are filled accordingly to your requirement.
(tested in GNU Awk 5.0.1)
GNU sed:
sed -zE 's/(.*\r?\n)(.*\r?\n?)/\2\1/' file
A Perl:
perl -0777 -pe 's/(.*\R.*\R)(.*\R.*\R?)/\2\1/' file
A ruby:
ruby -ne 'BEGIN{lines=[]}
lines<<$_
END{puts lines[2...4]+lines[0...2] }' file
Paste and awk:
paste -s file | awk -F'\t' '{print $3, $4, $1, $2}' OFS='\n'
A POSIX pipe:
paste -sd'\t\n' file | nl | sort -nr | cut -f 2- | tr '\t' '\n'
This seems to work:
awk -F'\n' '{print $3, $4, $1, $2}' OFS='\n' RS= ORS='\n\n' file.txt
Related
Basically the equivalent of "sort-by" method in CLI.
A sample input file (file1.txt):
one
three
five
eleven
thirteen
sixteen
Another input file (file2.txt, which lists length of corresponding line in file1.txt):
3
5
4
6
8
7
Desired output (sort lines in file1.txt by lines in file2.txt, in this case numerically; or in other words, sort lines in file1.txt by the line's length):
one
five
three
eleven
sixteen
thirteen
I've created a simple Perl script to do this. Sample usage:
% sort-by-lines file1.txt file2.txt
% sort-by-lines /etc/passwd <(perl -nE'say length' /etc/passwd)
But was wondering if a combination of more basic Unix commands (sort, cut, etc) can also do the same in a comparably simple fashion.
Are you just trying to sort a file by the length of each line? With standard tools in any shell on any UNIX box that'd be:
awk -v OFS='\t' '{print length(), NR, $0}' file | sort -k1,2n | cut -f3-
For example:
$ cat file
other stuff
text
foo
stuff
bar
$ awk -v OFS='\t' '{print length(), NR, $0}' file | sort -k1,2n | cut -f3-
foo
bar
text
stuff
other stuff
If that's not it then please edit your question to clarify what it is you're trying to do and what your actual question is.
Update - given the input you added to your question:
$ paste file2.txt file1.txt | sort -k1,2n | cut -f2-
one
five
three
eleven
sixteen
thirteen
Note that that won't necessarily preserve the order of lines of the same length - you'd need to add the GNU -s ("stable") option to sort to do that:
paste file2.txt file1.txt | sort -s -k1,2n | cut -f2-
or do this which is bash only:
paste file2.txt <(cat -n file1.txt) | sort -k1,2n | cut -f3-
or this which is portable to all shells/Unixes:
awk -v OFS='\t' 'NR==FNR{a[NR]=$0;next} {print a[FNR], FNR, $0}' file2.txt file1.txt | sort -k1,2n | cut -f3-
or do something with an explicit temp file or a here document.
I need to merge a single column from multiple CSV files whilst disregarding the headers.
file 1:
id,backer_uid,fname,lname
123,uj2uj2,JOHN,SMITH
file 2:
id,backer_uid,fname,lname
124,uj2uh3,BRIAN,DOOLEY
Output:
JOHN
BRIAN
Currently, I am using:
/*Merge 3rd column from all csv files*/
awk -F "\"*,\"*" '{print $3}’ *.csv >merged.csv
But how do I ignore the headers?
You can do it with awk, nearly as you have already done, by adding a condition on the FNR (the record number per file):
awk -F, 'FNR > 1 {print $3}' *.csv > merged.csv
Use tail and cut:
tail -q -n +2 *.csv | cut -f3 -d, > merged.csv
tail -n +2 prints all lines of files starting from line number 2
-q suppresses printing of file names
cut -f3 -d, extracts the third field, treating , as the delimiter
try: If you have to read only 2 files.
awk -F, 'FNR>1{print $(NF-1)}' file[12]
Here I am making field separator as comma and then checking if line number is greater than 1 then printing the second last field. Point to be noted here is file[12] will only read files named file1 and file2, if you have more than that files use file* then.
I have a file with the following structure (comma delimited)
116,1,89458180,17,FFFF,0403254F98
I want to add a blank column on the 4th field such that it becomes
116,1,89458180,,17,FFFF,0403254F98
Any inputs as to how to do this using awk or sed if possible ?
thank you
Assuming that none of the fields contain embedded commas, you can restate the task as replacing the third comma with two commas. This is just:
sed 's/,/,,/3'
With the example line from the file:
$ echo "116,1,89458180,17,FFFF,0403254F98" | sed 's/,/,,/3'
116,1,89458180,,17,FFFF,0403254F98
You can use this awk,
awk -F, '$4="," $4' OFS=, yourfile
(OR)
awk -F, '$4=FS$4' OFS=, yourfile
If you want to add 6th and 8th field,
awk -F, '{$4=FS$4; $1=FS$1; $6=FS$6}1' OFS=, yourfile
Through awk
$ echo '116,1,89458180,17,FFFF,0403254F98' | awk -F, -v OFS="," '{print $1,$2,$3,","$4,$5,$6}'
116,1,89458180,,17,FFFF,0403254F98
It prints a , after third field(delimited) by ,
Through GNU sed
$ echo 116,1,89458180,17,FFFF,0403254F98| sed -r 's/^([^,]*,[^,]*,[^,]*)(.*)$/\1,\2/'
116,1,89458180,,17,FFFF,0403254F98
It captures all the characters upto the third command and stored it into a group. Characters including the third , upto the last are stored into another group. In the replacement part, we just add an , between these two captured groups.
Through Basic sed,
Through Basic sed
$ echo 116,1,89458180,17,FFFF,0403254F98| sed 's/^\([^,]*,[^,]*,[^,]*\)\(.*\)$/\1,\2/'
116,1,89458180,,17,FFFF,0403254F98
echo 116,1,89458180,17,FFFF,0403254F98|awk -F',' '{print $1","$2","$3",,"$4","$5","$6}'
Non-awk
t="116,1,89458180,17,FFFF,0403254F98"
echo $(echo $t|cut -d, -f1-3),,$(echo $t|cut -d, -f4-)
You can use bellow awk command to achieve that.Replace the $3 with what ever the column that you want to make it blank.
awk -F, '{$3="" FS $3;}1' OFS=, filename
sed -e 's/\([^,]*,\)\{4\}/&,/' YourFile
replace the sequence of 4 [content (non comma) than comma ] by itself followed by a comma
Sorry title of this question is little confusing but I couldnt think of anything else.
I am trying to do something like this
cat fileA.txt | grep `awk '{print $1}'` fileB.txt
fileA contains 100 lines while fileB contains 100 million lines.
What I want is get id from fileA, grep that id in a different file-fileB and print that line.
e.g fileA.txt
1234
1233
e.g.fileB.txt
1234|asdf|2012-12-12
5555|asdd|2012-11-12
1233|fvdf|2012-12-11
Expected output is
1234|asdf|2012-12-12
1233|fvdf|2012-12-11
Getting rid of cat and awk altogether:
grep -f fileA.txt fileB.txt
awk alone can do that job well:
awk -F'|' 'NR==FNR{a[$0];next;}$1 in a' fileA fileB
see the test:
kent$ head a b
==> a <==
1234
1233
==> b <==
1234|asdf|2012-12-12
5555|asdd|2012-11-12
1233|fvdf|2012-12-11
kent$ awk -F'|' 'NR==FNR{a[$0];next;}$1 in a' a b
1234|asdf|2012-12-12
1233|fvdf|2012-12-11
EDIT
add explanation:
-F'|' #| as field separator (fileA)
'NR==FNR{a[$0];next;} #save lines in fileA in array a
$1 in a #if $1(the 1st field) in fileB in array a, print the current line from FileB
for further details I cannot explain here, sorry. for example how awk handle two files, what is NR and what is FNR.. I suggest that try this awk line in case the accepted answer didn't work for you. If you want to dig a little bit deeper, read some awk tutorials.
If the id's are on distinct lines you could use the -f option in grep as such:
cut -d "|" -f1 < fileB.txt | grep -F -f fileA.txt
The cut command will ensure that only the first field is searched for in the pattern searching using grep.
From the man page:
-f FILE, --file=FILE
Obtain patterns from FILE, one per line.
The empty file contains zero patterns, and therefore matches nothing.
(-f is specified by POSIX.)
input.txt
1,Ram,Fail
2,John,Fail
3,Ron,Success
param.txt (New Input)
1,Sam,Success
2,John,Sucess
Now i want to replace the whole line in input.txt with those present in param.txt .
1st column will act like a primary key.
Output.txt
1,Sam,Success
2,John,Sucess
3,Ron,Success
I tried as
awk 'FNR==NR{a[$1]=$2 FS $3;next}{ print $0, a[$1]}' input.txt param.txt > Output.txt
But it is merging the file contents.
This might work for you (GNU sed):
sed 's|^\([^,]*,\).*|/^\1/c\\&|' param.txt | sed -f - input.txt
Explanation:
Convert param.txt into a sed script using the first field as an address to change the line in the input.txt. s|^\([^,]*,\).*|/^\1/c\\&|
Run the script against the input.txt. sed -f - input.txt
This can be done with one call to sort:
sort -t, -k1,1n -us param.txt input.txt
Use a stable numerical sort on the first comma-delimited field, and list param.txt before input.txt so that the correct, newer, lines are preferred when eliminating duplicates.
You could use join(1) to make this work:
$ join -t, -a1 -j1 Input.txt param.txt | sed -E 's/,.*?,.*?(,.*?,.*?)/\1/'
1,Sam,Success
2,John,Sucess
3,Ron,Success
sed as a pipe tail strips fields from Input.txt out of replaced lines.
This will work only if both input files are sorted by first field.
Pure awk isn't really the right tool for the job. If you must use only awk, https://stackoverflow.com/a/5467806/1301972 is a good starting point for your efforts.
However, Unix provides some tools that will help with feeding awk the right input for what you're trying to do.
$ join -a1 -t, <(sort -n input.txt) <(sort -n param.txt) |
awk -F, 'NF > 3 {print $1 "," $4 "," $5; next}; {print}'
Basically, you're feeding awk a single file with the lines joined on the keys from input.txt. Then awk can parse out the fields you want for proper display or for redirection to your output file.
This should work in awk
awk -F"," 'NR==FNR{a[$1]=$0;next} ($1 in a){ print a[$1]; next}1' param.txt input.txt
Test:
$ cat input.txt
1,Ram,Fail
2,John,Fail
3,Ron,Success
$ cat param.txt
1,Sam,Success
2,John,Sucess
$ awk -F"," 'NR==FNR{a[$1]=$0;next} ($1 in a){ print a[$1]; next}1' param.txt input.txt
1,Sam,Success
2,John,Sucess
3,Ron,Success