I have two files. one has list of names (only one column) and the second file is with three columns with names, phone number, country.
What I want is to extract the data of the people whose names are not present in file 1, but only present in file2.
#!/bin/bash
for i in `cat file1 `
do
cat file2 | awk '{ if ($1 != "'$i'") {print $1 "\t" $2 "\t" $3 }}'>>NonResp
done
What I get is a weird result with more data than expected.
Kindly help.
You can do this with grep:
grep -v -F -f file1 file2
awk '{print $1}' file2 | comm -1 -3 file1 - | join file2 -
The files must already be sorted for this to work properly.
Explanation:
=> awk '{print $1}' file2 |
print only the first fileld of file2 and feed it to the next command (|)
=> comm -1 -3 file1 - |
compare file1 and the output of the last command (-) and suppress lines only in file1 (-1) as well as lines in both files (-3); that leaves lines in file2 only and feed this to the next command (|)
=> join file2 -
join the original file2 and the output from the last command (-) and write out the fields fo the matching lines (whitespace between fields is truncated, however)
Testcase:
cat <<EOF >file1
alan
bert
cindy
dave
fred
sunny
ted
EOF
cat <<EOF >file2
bert 01 AU
cindy 03 CZ
ginny 05 CN
ted 07 CH
zorro 09 AG
EOF
awk '{print $1}' file2 | comm -1 -3 file1 - | join file2 -
assuming the field delimiter as "," in file2
awk -F, 'FNR==NR{a[$1];next}!($1 in a)' file1 file2
if "," is not the delimiter ,then simply
awk 'FNR==NR{a[$1];next}!($1 in a)' file1 file2
would be sufficient.
Related
I have three files of name - File1, File2 and File3. The data of the three files is shown below:
File1:
// Class of "A2" of type "ONE".
// Class of "A3" of type "ONE".
// Class of "D1" of type "TWO".
// Class of "D2" of type "TWO".
// Class of "D3" of type "FOUR".
// Class of "D6" of type "FIVE."
File2:
#CLASS_NAMES = ("one",
"two",
"three",);
#CLASS_LIST_NAMES = ("ONE.A1",
"ONE.A2",
"ONE.A3",
"TWO.D1",
"TWO.D2");
File3:
D3
D4
D5
I need to check in File1 Class "D3" is present in the File2 of #CLASS_LIST_NAMES or not.
If it is not present in File2 of #CLASS_LIST_NAMES then I need to check in File3 if D3 is present there or not.
If D3 is present in File3 then the output should be as PASS and if it not present in both File2 and File3 the output should be FAIL.
Similarly I need to check for all the Class list-(A2, A3, D1, D2....) from File1 if they are present in the File2 of #CLASS_LIST_NAMES or not and if they are not present in File2, I need to verify with File3 and print the output as PASS or FAIL.
I tried the below code:
#!/bin/bash
sed -n '/#CLASS_LIST_NAMES =/,/)/p' File2
I'm stuck at here, can anyone tell me what need to be done next.
Deisred_Output: As from File1 - D6 is not found in both File2 and File3 it should print as FAIL. The output should be like below:
Fail: D6 is not found
You can achieve this with grep and awk
Use GNU grep which supports -P option
awk 'NR==FNR{a[$0]; next} !($0 in a){print "Fail: "$0 " is not found"}' <(cat file3 <(grep -Po '(?<=\.)[^"]+' file2)) <(grep -Po '(?<=of ")\w+' file1)
If you want to extract the classnames present only in the #CLASS_LIST_NAMES statement use below one.
awk 'NR==FNR{a[$0]; next} !($0 in a){print "Fail: "$0 " is not found"}' <(cat file3 <(sed -n '/#CLASS_LIST_NAMES/,/;$/p' | grep -Po '(?<=\.)[^"]+' file2)) <(grep -Po '(?<=of ")\w+' file1)
If the no of spaces in the file1 are not consistent, you can process using awk
# expects the 4th column is the variable, input format shouldn't change
awk 'NR==FNR{a[$0]; next} {gsub("\"","",$4)} !($4 in a){print "Fail: "$4" is not found"}' <(cat file3 <(sed -n '/#CLASS_LIST_NAMES/,/;$/p' | grep -Po '(?<=\.)[^"]+' file2)) file1
# alternate way using FPAT if the position of actual field can change, but it occurs first between double quotes
awk 'NR==FNR{a[$0]; next} {gsub("\"","",$1)} !($1 in a){print "Fail: "$1" is not found"}' <(cat file3 <(sed -n '/#CLASS_LIST_NAMES/,/;$/p' | grep -Po '(?<=\.)[^"]+' file2)) FPAT="\"[^ \"]+" file1
I need to get uniq lines when comparing 2 files. These files containing field separator ":" which should be treated as the end of line while comparing strings.
The file1 contains these lines
apple:tasty
apple:red
orange:nice
kiwi:awesome
kiwi:expensive
banana:big
grape:green
orange:oval
banana:long
The file2 contains these lines
orange:nice
banana:long
The output file should be (2 occurrences of orange and 2 occurrences of banana deleted)
apple:tasty
apple:red
kiwi:awesome
kiwi:expensive
grape:green
So the only strings before : should be compared
Is it possible to complete this task in 1 command ?
I tried to complete the task in such way but field separator does not work in that situation.
awk -F: 'FNR==NR {a[$0]++; next} !a[$0]' file1 file2 > outputfile
You basically had it, but $0 refers to the whole line when you want to deal with only the first field, which is $1.
Also you need to take care with the order of the input files. To use the values from file2 for deciding which lines to include from file1, process file2 first:
$ awk -F: 'FNR==NR {a[$1]++; next} !a[$1]' file2 file1
apple:tasty
apple:red
kiwi:awesome
kiwi:expensive
grape:green
One comment: awk is very ineffective with arrays. In real life with big files, better use something like:
comm -3 <(cut -d : -f 1 f1 | sort -u) <(cut -d : -f 1 f2 | sort -u) | grep -h -f /dev/stdin f1 f2
I have two files in my linux server.
File 1
9190784
9197256
9170546
9184139
9196854
File 2
S NO.,Column1,Column2,Column3
72070,9196854,TGM,AP
72071,9172071,BGM,MP
72072,9184139,AGM,KN
72073,9172073,TGM,AP
I want to write a script or a single line command in bash using awk command, so as whatever the element in File -1 should match the same with column 1 in File -2 and print Column 1, Column2 and Column3. Also if any entry is not found it should print entry from file 1 and print NA in Column 2 and Column 3
Output : it should redirect the output to a new file as below.
new_file
9190784,TGM,AP
9197256,NA,NA
9170546,NA,NA
9184139,AGM,KN
9196854,TGM,AP
I hope the query is understandable. Anyone please help me on the same.
standard join operation with awk
$ awk 'BEGIN {FS=OFS=","}
NR==FNR {a[$2]=$3 OFS $4; next}
{print $1, (($1 in a)?a[$1]:"NA" OFS "NA")} file2 file1
substring variation (not tested)
$ awk 'BEGIN {FS=OFS=","}
NR==FNR {a[substr($2,1,7)]=$3 OFS $4; next}
{key=substr($1,1,7);
print $1, ((key in a)?a[key]:"NA" OFS "NA")} file2 file1
Does it have to be awk? It's done with join:
Having two files:
echo '9190784
9197256
9170546
9184139
9196854' >file2
echo 'S NO.,Column1,Column2,Column3
72070,9196854,TGM,AP
72071,9172071,BGM,MP
72072,9184139,AGM,KN
72073,9172073,TGM,AP' > file1
One can join the on , as separator on the second field from the first file1 -12 with removed the first header line tail -n +2 and sorted using the second field sort -t, -k2 with the first field from the second file -21 sorted sort.
join -t, -12 -21 -o1.2,1.3,1.4 <(tail -n +2 file1 | sort -t, -k2) <(sort file2)
will output:
9184139,AGM,KN
9196854,TGM,AP
I want to compare the second column (delimited by a whitespace) in file1:
n01443537/n01443537_481.JPEG n01443537
n01629819/n01629819_420.JPEG n01629819
n02883205/n02883205_461.JPEG n02883205
With the second column (delimited by a whitespace) in file2:
val_8447.JPEG n09256479
val_68.JPEG n01443537
val_1054.JPEG n01629819
val_1542.JPEG n02883205
val_8480.JPEG n03089624
If there is a match, I would like to print out the corresponding line of file2.
Desired output in this example:
val_68.JPEG n01443537
val_1054.JPEG n01629819
val_1542.JPEG n02883205
I tried the following, but the output file is empty:
awk -F' ' 'NR==FNR{c[$2]++;next};c[$2] > 0' file1.txt file2.txt > file3.txt
Also tried this, but the result was the same (empty output file):
awk 'NR==FNR{a[$2];next}$2 in a' file1 file2 > file3.txt
GNU join exists for this purpose.
join -o "2.1 2.2" -j 2 <(sort -k 2 file1) <(sort -k 2 file2)
Using awk:
awk 'FNR==NR{a[$NF]; next} $NF in a' file1 file2
val_68.JPEG n01443537
val_1054.JPEG n01629819
val_1542.JPEG n02883205
Here is a grep alternative with process substitution:
grep -f <(awk '{print " " $NF "$"}' file1) file2
Using print " " $NF "$" to create a regex like " n01443537$" so that we match only last column in grep.
Here is what I am trying to do.
File1:
abc
bcd
cde
def
efg
fgh
ghi
File2:
ip:/vol0/scratch/&
ip:/vol0/sysbuild/
ip:/vol0/cde
ip:/vol0/mnt/cm/&
ip:/vol0/&
ip:/vol0/mnt/fgh
ip:/vol0/mnt/&
As you can see File2 has & at the end of some lines, I need to replace the & with corresponding line in File1 and ignore the lines without the & For example, if line 2 and line 3 doesn't have & the script would skip line 2 and 3 in both files and go to line 4 to replace the &
How would I achieve this with shell script.
Using paste and awk:
$ paste file2 file1 | awk 'sub(/&\s+/,"")'
ip:/vol0/scratch/abc
ip:/vol0/mnt/cm/def
ip:/vol0/efg
ip:/vol0/mnt/ghi
Wasn't 100% clear if you wanted the lines not ending in & in the output:
$ paste file2 file1 | awk '{sub(/&\s+/,"");print $1}'
ip:/vol0/scratch/abc
ip:/vol0/sysbuild/
ip:/vol0/cde
ip:/vol0/mnt/cm/def
ip:/vol0/efg
ip:/vol0/mnt/fgh
ip:/vol0/mnt/ghi
With sed:
$ paste file2 file1 | sed -rn '/&/s/&\s+//p'
ip:/vol0/scratch/abc
ip:/vol0/mnt/cm/def
ip:/vol0/efg
ip:/vol0/mnt/ghi
awk 'NR==FNR{a[NR]=$0;next} sub(/&/,a[FNR])' file1 file2
paste file1 file2 | awk 'gsub( /&/, $1 )' | cut -f2-
try this
awk '{if (NR == FNR){f[NR]= $0;}else {gsub("&",f[FNR],$0); print $0}}' file1.txt file2.txt
This might work for you (GNU sed):
sed = file1 | sed -r 'N;s/(.*)\n(.*)/\1s|\&$|\2|/' | sed -f - file2
sed = file1 generate line numbers
sed -r 'N;s/(.*)\n(.*)/\1s|\&$|\2|/' combine line number with data line and produce a sed substitution command using the line number as an address.
sed -f - file2 feed the above commands into a sed invocation using the -f switch and the standard input -