foo.txt consists of
printf("%f \n\n",row1.req_pnttime);
printf("%f \n\n",avinash);
printf("%f \n\n",foo);
printf("%f \n\n",bar);
bar.txt consists of,
foo
bar
foo1
bar1
I want to combine first line of bar.txt with the first line of foo.txt in a specified place.Same for all the lines.Like below,
Expected output:
printf("foo%f \n\n",row1.req_pnttime);
printf("bar%f \n\n",avinash);
printf("foo1%f \n\n",foo);
printf("bar1%f \n\n",bar);
I tried the below , but it won't work.
awk -v FS="\"" -v OFS="\"" 'FNR==NR{a=$0;}{$2=a[FNR]$2}1' bar.txt foo.txt
You can try this,
sed 'R bar.txt' foo.txt | sed 'N;s/^\(.*\)\(%.*\)\n\(.*\)/\1\3\2/'
Test:
sat:~# sed 'R bar.txt' foo.txt | sed 'N;s/^\(.*\)\(%.*\)\n\(.*\)/\1\3\2/'
printf("foo%f \n\n",row1.req_pnttime);
printf("bar%f \n\n",avinash);
printf("foo1%f \n\n",foo);
printf("bar1%f \n\n",bar);
Ok, i figured out the problem. One through awk,
$ awk -v FS="\"" -v OFS="\"" 'FNR==NR{a[FNR]=$0;next}{$2=a[FNR]$2}1' bar.txt foo.txt
printf("foo%f \n\n",row1.req_pnttime);
printf("bar%f \n\n",avinash);
printf("foo1%f \n\n",foo);
printf("bar1%f \n\n",bar);
Assuming that the text is always aligned to the same column you could do this:
cut -b1-8 foo.txt > a
cut -b9- foo.txt > b
paste -d '' a bar.txt b
Related
Here are two files where I need to eliminate the data that they do not have in common:
a.txt:
hello world
tom tom
super hero
b.txt:
hello dolly 1
tom sawyer 2
miss sunshine 3
super man 4
I tried:
grep -f a.txt b.txt >> c.txt
And this:
awk '{print $1}' test1.txt
because I need to check only if the first word of the line exists in the two files (even if not at the same line number).
But then what is the best way to get the following output in the new file?
output in c.txt:
hello dolly 1
tom sawyer 2
super man 4
Use awk where you iterate over both files:
$ awk 'NR == FNR { a[$1] = 1; next } a[$1]' a.txt b.txt
hello dolly 1
tom sawyer 2
super man 4
NR == FNR is only true for the first file making { a[$1] = 1; next } only run on said file.
Use sed to generate a sed script from the input, then use another sed to execute it.
sed 's=^=/^=;s= .*= /p=' a.txt | sed -nf- b.txt
The first sed turns your a.txt into
/^hello /p
/^tom /p
/^super /p
which prints (p) whenever a line contains hello, tom, or super at the beginning of line (^) followed by a space.
This combines grep, cut and sed with process substitution:
$ grep -f <(cut -d ' ' -f 1 a.txt | sed 's/^/^/') b.txt
hello dolly 1
tom sawyer 2
super man 4
The output of the process substitution is this (piping to cat -A to show spaces):
$ cut -d ' ' -f 1 a.txt | sed 's/^/^/;s/$/ /' | cat -A
^hello $
^tom $
^super $
We then use this as input for grep -f, resulting in the above.
If your shell doesn't support process substitution, but your grep supports reading from stdin with the -f option (it should), you can use this instead:
$ cut -d ' ' -f 1 a.txt | sed 's/^/^/;s/$/ /' | grep -f - b.txt
hello dolly 1
tom sawyer 2
super man 4
I'm trying to use sed to remove the last occurrence of } from a file. So far I have this:
sed -i 's/\(.*\)}/\1/' file
But this removes as many } as there are on the end of the file. So if my file looks like this:
foo
bar
}
}
}
that command will remove all 3 of the } characters. How can I limit this to just the last occurrence?
someone game me this as a solution
sed -i '1h;1!H;$!d;g;s/\(.*\)}/\1/' file
I'm just not sure it's as good as the above awk solution.
sed is an excellent tool for simple substitutions on a single line. For anything else, just use awk, e.g. with GNU awk for gensub() and multi-char RS:
$ cat file1
foo
bar
}
}
}
$
$ cat file2
foo
bar
}}}
$
gawk -v RS='^$' -v ORS= '{$0=gensub(/\n?}([^}]*)$/,"\\1","")}1' file1
foo
bar
}
}
$
$ gawk -v RS='^$' -v ORS= '{$0=gensub(/\n?}([^}]*)$/,"\\1","")}1' file2
foo
bar
}}
$
Note that the above will remove the last } char AND a preceding newline if present as I THINK that's probably what you would actually want but if you want to ONLY remove the } and leave a trailing newline in those cases (as I think all of the currently posted sed solutions would do), then just get rid of \n? from the matching RE:
$ gawk -v RS='^$' -v ORS= '{$0=gensub(/}([^}]*)$/,"\\1","")}1' file1
foo
bar
}
}
$
And if you want to change the original file without manually specifying a tmp file, just use the -i inplace argument:
$ gawk -i inplace -v RS='^$' -v ORS= '{$0=gensub(/}([^}]*)$/,"\\1","")}1' file1
$ cat file1
foo
bar
}
}
$
With a buffer you can modify the file directly:
awk 'BEGIN{file=ARGV[1]}{a[NR]=$0}/}/{skip=NR}END{for(i=1;i<=NR;++i)if(i!=skip)print a[i]>file}' file
thnks to #jthill for remark for the 1 line file issue
sed ':a
$ !{N
ba
}
$ s/}\([^}]*\)$/\1/' YourFile
Need to load the file in buffer first. This does not remove the new line if } is alone on a line
When I read "do something with the last ...", I think "reverse the file, do something with the first ..., re-reverse the file"
tac file | awk '!seen && /}/ {$0 = gensub(/(.*)}/, "\\\1", 1); seen = 1} 1' | tac
I have 3 different files.
Test1.txt , Test2.txt & Test3.txt
Test1.txt contains
JJTP#yahoo.com
BBMU#ssc.com
HK#glb.com
Test2.txt contains
SFTY#gmail.com
JJTP#yahoo.com
Test3.txt contains
JJTP#yahoo.com
HK#glb.com
I would like to see only matching records in these 3 files.
so the matching records in above example will be JJTP#yahoo.com
The output should be
JJTP#yahoo.com
If you don't have duplicate lines in each file then:
$ awk '++a[$1]==3' test[1-3]
JJTP#yahoo.com
Here is an awk that has a mix of jaypal and sudo_o solution.
It will not give false positive since it test for uniqueness of the lines.
awk '!a[$1 FS FILENAME]++ && ++b[$1]==3' test*
JJTP#yahoo.com
If you have a unknown number of files, this could be an option
awk '!a[$1 FS FILENAME]++ && ++b[$1]==ARGC-1' test*
The ARGC store the number of files read by awk + 1
comm lists common lines for two files. Just find the common lines in the first two files, then pipe the output to comm again and find the common lines with the third file.
comm -12 <(sort Test1.txt) <(sort Test2.txt) | comm -12 - <(sort Test3.txt)
Here is how you'd do with awk:
awk '
FILENAME == ARGV[1] { a[$0]++ }
FILENAME == ARGV[2] && ($0 in a) { b[$0]++ }
FILENAME == ARGV[3] && ($0 in b)' file1 file2 file3
Output:
JJTP#yahoo.com
To find the common lines in two files, you can use:
sort Test1.txt Test2.txt | uniq -d
Or, if you wish to preserve the order found in Test1.txt, you may use:
while read x; do grep -w "$x" Test2.txt; done < Test1.txt
For three files, repeat this:
sort Test1.txt Test2.txt | uniq -d | sort - Test3.txt | uniq -d
Or:
cat Test1.txt |\
while read x; do grep -w "$x" Test2.txt; done |\
while read x; do grep -w "$x" Test3.txt; done
The sort method assumes that the files themselves don't have duplicate lines; if so, you may need create temporary files.
If you wish to use sed rather than grep, try sed -n "/^$x$/p".
Is there a unix one liner to do this?
head -n 3 test.txt > out_dir/test.head.txt
grep hello test.txt > out_dir/test.tmp.txt
cat out_dir/test.head.txt out_dir/test.tmp.txt > out_dir/test.hello.txt
rm out_dir/test.head.txt out_dir/test.tmp.txt
I.e., I want to get the header and some grep lines from a given file, simultaneously.
Use awk:
awk 'NR<=3 || /hello/' test.txt > out_dir/test.hello.txt
You can say:
{ head -n 3 test.txt ; grep hello test.txt ; } > out_dir/test.hello.txt
Try using sed
sed -n '1,3p; /hello/p' test.txt > out_dir/test.hello.txt
The awk solution is the best, but I'll add a sed solution for completeness:
$ sed -n test.txt -e '1,3p' -e '4,$s/hello/hello/p' test.txt > $output_file
The -n says not to print out a line unless specified. The -e are the commands '1,3p prints ou the first three lines 4,$s/hello/hello/p looks for all lines that contain the word hello, and substitutes hello back in. The p on the end prints out all lines the substitution operated upon.
There should be a way of using 4,$g/HELLO/p, but I couldn't get it to work. It's been a long time since I really messed with sed.
Of course, I would go awk but here is an ed solution for the pre-vi nostalgics:
ed test.txt <<%
4,$ v/hello/d
w test.hello.txt
%
Here is what I am trying to do.
File1:
abc
bcd
cde
def
efg
fgh
ghi
File2:
ip:/vol0/scratch/&
ip:/vol0/sysbuild/
ip:/vol0/cde
ip:/vol0/mnt/cm/&
ip:/vol0/&
ip:/vol0/mnt/fgh
ip:/vol0/mnt/&
As you can see File2 has & at the end of some lines, I need to replace the & with corresponding line in File1 and ignore the lines without the & For example, if line 2 and line 3 doesn't have & the script would skip line 2 and 3 in both files and go to line 4 to replace the &
How would I achieve this with shell script.
Using paste and awk:
$ paste file2 file1 | awk 'sub(/&\s+/,"")'
ip:/vol0/scratch/abc
ip:/vol0/mnt/cm/def
ip:/vol0/efg
ip:/vol0/mnt/ghi
Wasn't 100% clear if you wanted the lines not ending in & in the output:
$ paste file2 file1 | awk '{sub(/&\s+/,"");print $1}'
ip:/vol0/scratch/abc
ip:/vol0/sysbuild/
ip:/vol0/cde
ip:/vol0/mnt/cm/def
ip:/vol0/efg
ip:/vol0/mnt/fgh
ip:/vol0/mnt/ghi
With sed:
$ paste file2 file1 | sed -rn '/&/s/&\s+//p'
ip:/vol0/scratch/abc
ip:/vol0/mnt/cm/def
ip:/vol0/efg
ip:/vol0/mnt/ghi
awk 'NR==FNR{a[NR]=$0;next} sub(/&/,a[FNR])' file1 file2
paste file1 file2 | awk 'gsub( /&/, $1 )' | cut -f2-
try this
awk '{if (NR == FNR){f[NR]= $0;}else {gsub("&",f[FNR],$0); print $0}}' file1.txt file2.txt
This might work for you (GNU sed):
sed = file1 | sed -r 'N;s/(.*)\n(.*)/\1s|\&$|\2|/' | sed -f - file2
sed = file1 generate line numbers
sed -r 'N;s/(.*)\n(.*)/\1s|\&$|\2|/' combine line number with data line and produce a sed substitution command using the line number as an address.
sed -f - file2 feed the above commands into a sed invocation using the -f switch and the standard input -