Oracle merge into with text similarlity - oracle

Question

I have 2 tables: from_country and to_country. I want to bring new records and update records to to_country
Definition and data
--
CREATE TABLE from_country
(
country_code varchar2(255) not null
);
--
CREATE TABLE to_country
(
country_code varchar2(255) not null
);
-- Meaning match
INSERT INTO from_country
(country_code)
VALUES
('United States of America');
-- Match 100%
INSERT INTO from_country
(country_code)
VALUES
('UGANDA');
-- Meaning match, but with domain knowledge
INSERT INTO from_country
(country_code)
VALUES
('CON CORRECT');
-- Brand new country
INSERT INTO from_country
(country_code)
VALUES
('NEW');
--
INSERT INTO to_country
(country_code)
VALUES
('USA');
-- Match 100%
INSERT INTO to_country
(country_code)
VALUES
('UGANDA');
-- Meaning match, but with domain knowledge
INSERT INTO to_country
(country_code)
VALUES
('CON');
I need to run merge into so I bring data from from_county to to_country
Here is my 1st attempt, but it only does a equal, which is not good enough. I need some smartness so that it is able to do meaning match.
If anyone know how to do it, please provide your solution.
merge into
to_country to_t
using
from_country from_t
on
(to_t.country_code = from_t.country_code)
when not matched then insert (
country_code
)
values (
from_t.country_code
);
So in a nutshell, here is what I want
from_table:
United States of America
UGANDA
CON CORRECT
NEW
to_table:
USA
UGANDA
CON
After oracle merge into
the new to_country table:
United States of America
UGANDA
CON CORRECT
NEW
sql fiddle: http://sqlfiddle.com/#!4/f512d
Please note that this is my simplified example. I have larger data set.

Answer 1

Since the match is not guaranteed unique, you have to write a query that will return only one match using some decision.
Here is a simplified case which uses a naive match and then just picks one value when there is more than one match:
merge into to_country t
using (
select * from (
select t.rowid as trowid
,f.country_code as fcode
,t.country_code as tcode
,case when t.country_code is null then 1 else
row_number()
over (partition by t.country_code
order by f.country_code)
end as match_no
from from_country f
left join to_country t
on f.country_code like t.country_code || '%'
) where match_no = 1
) s
on (s.trowid = t.rowid)
when matched then update set country_code = s.fcode
when not matched then insert (country_code) values (s.fcode);
Result in to_country:
USA
UGANDA
CON CORRECT
United States of America
Now that that's taken care of, you just need to make the match algorithm smarter. This is where you need to look at the whole dataset to see what sort of errors there are - i.e. typos, etc.
You could try some of the procedures in Oracle's supplied UTL_MATCH for this purpose: https://docs.oracle.com/cd/E18283_01/appdev.112/e16760/u_match.htm - such as EDIT_DISTANCE, or JARO_WINKLER.
Here is an example using the Jaro Winkler algorithm:
merge into to_country t
using (
select * from (
select t.rowid as trowid
,f.country_code as fcode
,t.country_code as tcode
,case when t.country_code is null then 1
else row_number() over (
partition by t.country_code
order by utl_match.jaro_winkler_similarity(f.country_code,t.country_code) desc)
end as match_no
from from_country f
left join to_country t
on utl_match.jaro_winkler_similarity(f.country_code,t.country_code) > 70
) where match_no = 1
) s
on (s.trowid = t.rowid)
when matched then update set country_code = s.fcode
when not matched then insert (country_code) values (s.fcode);
SQL Fiddle: http://sqlfiddle.com/#!4/f512d/23
Note that I've picked an arbitrary cutoff of >70%. This is because UGANDA vs. USA has a Jaro Winkler similarity of 70.
This results in the following:
United States of America
USA
UGANDA
CON NEW
To see how these algorithms fare, run something like this:
select f.country_code as fcode
,t.country_code as tcode
,utl_match.edit_distance_similarity(f.country_code,t.country_code) as ed
,utl_match.jaro_winkler_similarity(f.country_code,t.country_code) as jw
from from_country f
cross join to_country t
order by 2, 4 desc;
FCODE TCODE ED JW
======================== ====== === ===
CON NEW CON 43 86
CON CORRECT CON 28 83
UGANDA CON 17 50
United States of America CON 0 0
UGANDA UGANDA 100 100
United States of America UGANDA 9 46
CON NEW UGANDA 15 43
CON CORRECT UGANDA 0 41
UGANDA USA 34 70
United States of America USA 13 62
CON CORRECT USA 0 0
CON NEW USA 0 0
SQL Fiddle: http://sqlfiddle.com/#!4/f512d/22