How to efficiently transform narrow table into wide in Oracle? - oracle

Question

I have a narrow table with the following columns:
<Customer ID> <Field ID> <Value>, all of them are numbers.
I want to reshape this table into the wide format:
<Customer ID> <Field1> <Field2> <Field3> ...
I have a separate dictionary table DIC_FIELDS that translates Field ID into Field Name.
I work on EXADATA server. The narrow table has 2.5 billion records, and we have about 200 fields.
The obvious simple solution below badly fills up all temporary space on our EXADATA server.
create table WIDE_ADS as (
CUSTOMERID
,max(case when FIELDID = 1 then VALUE end) as GENDER
,max(case when FIELDID = 2 then VALUE end) as AGE
,max(case when FIELDID = 3 then VALUE end) as EDUCATION
from NARROW_ADS
group by CUSTOMERID
);
We tried also a cleverer and manual method:
create index index1
on SZEROKI_ADS(CUSTOMERID);
DECLARE
rowidWide rowid;
type tColNames is table of STRING(32000) index by pls_integer ;
arrColNames tColNames;
x_CustomerID number;
strColName varchar2(32);
strColvalue varchar2(32000);
strSQL varchar2(200);
lngCounter pls_integer;
lngFieldID pls_integer;
BEGIN
lngCounter := 0;
-- we pre-load the dictionary arrColNames to speedup lookup.
for DIC_EL in (select * from DIC_FIELDS order by FIELDID) LOOP
lngFieldID := to_number(DIC_EL.FIELDID);
arrColNames(lngFieldID) := DIC_EL.FIELDNAME;
END LOOP;
FOR NARROW_REC IN (SELECT * FROM NARROW_ADS where VALUE is not null ) LOOP
strColName := arrColNames(NARROW_REC.FIELDID);
strColvalue := NARROW_REC.VALUE;
x_IDKlienta := NARROW_REC.CUSTOMERID;
BEGIN
select rowid into rowidWide from WIDE_ADS
where CUSTOMERID = NARROW_REC.CUSTOMERID;
strSQL := 'update :1 set :2 = :3 where rowid = :4';
execute immediate strSQL using WIDE_ADS, strColName, strColvalue, rowidWide;
EXCEPTION
WHEN NO_DATA_FOUND THEN
strSQL :=
'insert into '|| WIDE_ADS ||' (CUSTOMERID, '|| strColName ||')
values
(:1, :2)';
execute immediate strSQL using x_CustomerID, to_number(strColvalue) ;
END;
IF lngCounter=10000 THEN
COMMIT;
lngCounter:=0;
dbms_output.put_line('Clik...');
ELSE
lngCounter:=lngCounter+1;
END IF;
END LOOP;
END;
Although it doesn't take a temp, it fails miserably performance-wise; it processes 10 000 records in 50 sec - that is about 1000 times slower, then expected.
What can we do to speed up the process?

Answer 1

As Lalit comments, try to do it in chunks based on CUSTOMERID.
First, create a index on CUSTOMERID (if it does not exist):
CREATE INDEX INDNARROWADS ON NARROW_ADS(CUSTOMERID);
Second, we are going to create an auxiliary table to compute buckets based on CUSTOMERID (in this example we create 1000 buckets, 1 bucket will represent 1 block insert statement):
CREATE TABLE BUCKETS(MINCUSTOMER, MAXCUSTOMER, BUCKETNUM) AS
SELECT MIN(CUSTOMERID), MAX(CUSTOMERID), BUCKET
FROM (SELECT CUSTOMERID,
WIDTH_BUCKET(CUSTOMERID,
(SELECT MIN(CUSTOMERID) FROM NARROW_ADS),
(SELECT MAX(CUSTOMERID) FROM NARROW_ADS),
1000) BUCKET
FROM NARROW_ADS)
GROUP BY BUCKET;
You can use more/less buckets modifying the fourth argument of WIDTH_BUCKET function.
Third, create the WIDE_ADS table (the structure with no data). You should do it manually (with special attention on storage parameters) but you can also use your own query with a WHERE false condition:
create table WIDE_ADS as select
CUSTOMERID
,max(case when FIELDID = 1 then VALUE end) as GENDER
,max(case when FIELDID = 2 then VALUE end) as AGE
,max(case when FIELDID = 3 then VALUE end) as EDUCATION
from NARROW_ADS
where 1=0;
Fourth, execute your query over each bucket (1 bucket means 1 insert statement):
BEGIN
FOR B IN (SELECT * FROM BUCKETS ORDER BY BUCKETNUM) LOOP
INSERT INTO WIDE_ADS
SELECT
CUSTOMERID
,max(case when FIELDID = 1 then VALUE end) as GENDER
,max(case when FIELDID = 2 then VALUE end) as AGE
,max(case when FIELDID = 3 then VALUE end) as EDUCATION
FROM NARROW_ADS
WHERE CUSTOMERID BETWEEN B.MINCUSTOMER AND B.MAXCUSTOMER
GROUP by CUSTOMERID;
COMMIT;
END LOOP;
END;
And finally, drop auxiliary table (and index if it is not necessary).
Oracle optimizer should use the index on CUSTOMERID to perform an "index range scan" over NARROW_ADS. So, each INSERT should find efficiently its corresponding interval.
Note that WIDTH_BUCKETS creates buckets based on uniform divisions over the specified interval on CUSTOMERID (from min to max values). It does not create buckets based on uniform number of rows. And also note that NARROW_ADS must not be modified while this process is being executed.
As the PL/SQL block executes a COMMIT on each iteration and the loop iterates over buckets using the BUCKETNUM order, you can see how WIDE_ADS grows and which bucket is being processed (retrieving the max CUSTOMERID from WIDE_ADS and find its corresponding bucket on BUCKETS table).
If temporary space usage is to high, then increase the number of buckets (each insert will be smaller).