How do i import a script with 3954275 Lines of Insert Statements into a Oracle 10g. I can do it with sqlplus user/pass # script.sql but this is dam slow (even worse the commit is at the end of this 900MB file. I dont know if my Oracle configuration can handle this). Is there a better (faster) way to import the Data?
Btw. the DB is empty before the Import.
Use SQL*Loader.
It can parse even your INSERT commands if you don't have your data in another format.
SQL*Loader is a good alternative if your 900MB file contains insert statements to the same table. It will be cumbersome if it contains numerous tables. It is the fastest option however.
If for some reason a little improvement is good enough, then make sure your sessions CURSOR SHARING parameter is set to FORCE or SIMILAR. Each insert statement in your file will likely be the same except for the values. If CURSOR_SHARING is set to EXACT, then each of insert statements needs to be hard parsed, because it is unique. FORCE and SIMILAR automatically turns your literals in the VALUES clause to bind variables, removing the need for the hard parse over and over again.
You can use the script below to test this:
set echo on
alter system flush shared_pool
/
create table t
( id int
, name varchar2(30)
)
/
set echo off
set feedback off
set heading off
set termout off
spool sof11.txt
prompt begin
select 'insert into t (id,name) values (' || to_char(level) || ', ''name' || to_char(level) || ''');'
from dual
connect by level <= 10000
/
prompt end;;
prompt /
spool off
set termout on
set heading on
set feedback on
set timing on
#sof11.txt
set timing off
alter session set cursor_sharing = force
/
set timing on
#sof11.txt
set timing off
alter session set cursor_sharing = exact
/
set echo off
drop table t purge
/
The example executes 10,000 statements like "insert into t (id,name) values (1, 'name1');
". The output on my laptop:
SQL> alter system flush shared_pool
2 /
Systeem is gewijzigd.
SQL> create table t
2 ( id int
3 , name varchar2(30)
4 )
5 /
Tabel is aangemaakt.
SQL> set echo off
PL/SQL-procedure is geslaagd.
Verstreken: 00:00:17.10
Sessie is gewijzigd.
PL/SQL-procedure is geslaagd.
Verstreken: 00:00:05.50
More than 3 times as fast with CURSOR_SHARING set to FORCE.
Hope this helps.
Regards,
Rob.
Agreed with the above: use SQL*Loader.
However, if that is not an option, you can adjust the size of the blocks that SQL Plus brings in by putting the statement
SET arraysize 1000;
at the beginning of your script. This is just an example from my own scripts, and you may have to fine tune it to your needs considering latency, etc. I think it defaults to like 15, so you're getting a lot of overhead in your script.
Related
My query is the following :
ALTER TABLE EMPLOYEE MODIFY LOB(DOCUMENT) (SHRINK SPACE);
The parallel degree for the table is 1 :
SELECT DEGREE FROM USER_TABLES WHERE TABLE_NAME = 'EMPLOYEE';
Result : 1
Table EMPLOYEE is not partitioned.
If I launch the same query with a parallel degree, the system does not complain, but is it silently ignored ?
ALTER TABLE EMPLOYEE MODIFY LOB(DOCUMENT) (SHRINK SPACE) PARALLEL 8;
Any chance that the query will be faster ?
DDL operations will run in parallel when:
The degree of the table is not 1
OR
The session parallel ddl has been enabled by alter session enable parallel ddl
Anyway, you should always run the alter session enable parallel ddl before running any DDL operation that can run in parallel. Although the documentation did not say that shrink can run in parallel, the syntax is allowed, so I guess you can test whether it runs faster or not.
The parallel DDL statements for nonpartitioned tables and indexes are:
CREATE INDEX
CREATE TABLE AS SELECT
ALTER INDEX REBUILD
The parallel DDL statements for partitioned tables and indexes are:
CREATE INDEX
CREATE TABLE AS SELECT
ALTER TABLE {MOVE|SPLIT|COALESCE} PARTITION
ALTER INDEX {REBUILD|SPLIT} PARTITION
Example
SQL> create table t ( c1 clob ) ;
Table created.
SQL> alter table t MODIFY LOB(c1) (SHRINK SPACE) PARALLEL 8 ;
Table altered.
Instead of SHRINK you can always move the segment lob, which I can assure 100% will run in parallel and faster. The problem is that if you have indexes, there will become invalid.
UPDATE
To move the lob segment, you must do the following
Moving LOB:
SQL> spoolmovelob.sql
SET HEADING OFF
SET pagesize 200
SET linesize 200
select 'ALTER TABLE <owner>.'||TABLE_NAME||' MOVE LOB('||COLUMN_NAME||') STORE AS (TABLESPACE <Tablespace_name>) parallel 5 nologging;' from dba_lobs where TABLESPACE_NAME='<Tablespace_name>';
Note: The above query will include all the LOB,LOBSEGMENT,LOBINDEXES
Moving Table:
SQL> spool /home/oracle/moveTables.sql
SET HEADING OFF
SET PAGESIZE 200
SET LINESIZE 200
select ' ALTER TABLE <owner>.'||TABLE_NAME||' MOVE TABLESPACE <Tablespace_name>) parallel 5 nologging;' from dba_tables where owner='<owner name>';
Moving Indexes:
SQL> spool /home/oracle/moveIndex.sql
SET HEADING OFF
SET long 9999
SET linesize 200
select 'alter index <owner>.'||index_name||' from dba_indexes 'rebuild tablespace <Tablespace_name>) online parallel X nologging;' where owner='<owner>.';
Remember to replace X with the specific degree.
Did you enable parallel ddl?
alter session force parallel ddl parallel 8
Show us your session parameters:
select *
from v$ses_optimizer_env e
where e.sid=userenv('sid')
and (
name like '%parallel%'
or name like '%cpu%'
or name like '%optim%'
)
order by name
Accroding to the Oracle documentation
the ALTER TABLE MODIFY operation cannot be run in parallel.
If I launch the same query with a parallel degree, the system does not complain, but is it silently ignored ?
Depending on how your database in general and your specific seession are configured, it may be ignored.
Any chance that the query will be faster ?
This will very much depend on your specific dataset, your available system resources, and how you set up your parallelism. Pay attention to the parallel_degree_policy setting; it controls default behavior.
See the Oracle whitepaper, "Parallel Execution with Oracle Database" for a more complete understanding. In particular, read the section on "Controlling Parallel Execution" beginning on page 21.
I'm spooling from a SQL*Plus script many SELECTs to separate CSVs, most of which JOIN on the same SELECT, which I've placed as a WITH before the SELECT in each SPOOL in order to at least be able to find&replace easily.
Now I'm trying to rid the script of the code repetition and to optimize for performance by maybe storing the results of the repeated SELECT and reusing them in each SPOOL.
I've tried using variables to store the results of the SELECT that is repeated as WITH, but I can't seem to be able to mix SQL*Plus SPOOLs with PL/SQL code (using SPOOL inside DECLARE BEGIN/END block fails with only the number "28" being output on the command-line). My first thought was something like temporary tables were needed, but there's only read-only access, so no DDL like CREATE/ALTER, etc.
SET MARKUP CSV ON
-- omitting delimiter and other SQL*Plus options set
SPOOL C:\test\1.csv
WITH Y AS ( SELECT * FROM Z )
SELECT * FROM X JOIN Y ON Y.W = X.W
;
SPOOL OFF
SPOOL C:\test\2.csv
WITH Y AS ( SELECT * FROM Z )
SELECT * FROM W JOIN Y ON Y.X = W.X
;
SPOOL OFF
-- many other different spools joining on the same WITH SELECT
exit;
This code works fine, but I'd prefer to not have to repeat the same WITH as it kills performance and makes the code a chore to maintain / work on. However, the main concern here is the repetition of code and not necessarily the performance hit, at least for now, although it is an ugly thing to have the server repeat the same queries over and over.
Obviously, I'm not literate in Oracle / PL/SQL.
It would help if you (or someone else) could create a view as
create view v_y
as
SELECT * FROM Z --> this is your "Y" CTE
I presume that SELECT * FROM Z is more complex than this simple statement so - if nothing else - your code would look prettier. See if a DBA or such could create a view for you.
It also means that you'd avoid CTE completely and join tables to the v_y view directly.
... fails with only the number "28" being output on the command-line
It means that you should terminate that PL/SQL block with a slash. Or, simply enter slash now and press enter:
28 / --> now press enter
Anyway, SPOOL - which is a SQL*Plus command - won't work in PL/SQL procedures. But, you could create a file using UTL_FILE package. It isn't as simple as SPOOL as you should be granted to use that package, DBA should create a directory (Oracle object) which points to a directory on database server's hard disk, grant you privileges to access it (the directory). That is far from being impossible, but you can't do it yourself.
Perhaps you'd rather ask them to create that view for you, eh?
To give an answer to the following:
but I can't seem to be able to mix SQL*Plus SPOOLs with PL/SQL code
(using SPOOL inside DECLARE BEGIN/END block fails with only the number
"28" being output on the command-line)
You can use the Spool and PL/SQL block using the following technique:
-- test.sql file
spool tejash_1.txt
declare
cursor c_emp is
select seq, req from table1;
begin
for r_emp in c_emp loop
dbms_output.put_line(r_emp.seq || ' ' || r_emp.req);
end loop;
end;
/
spool off
-- end of test.sql file
-- Command to execute in SQL*Plus
SQL> set serverout on
SQL> SET echo off
SQL> SET feedback off
SQL> SET term off
SQL> #test.sql
-- output in tejash_1.txt file
001 X1
002 X1
003 X1
004 X1
-- End of output in tejash_1.txt file
Cheers!!
In an attempt to more easily view the output in SQLPlus, I fiddled with window-size properties and font(tiniest font).
But this is what I get :
It is exactly the same, just much less readable.
I just want to run the query :
select * from user_tables where rownum < 10 ;
Without the messy unreadable lines (currently I can't tell apart the data from table headings)
With the help of one useful SQL*Plus option and one additional helper option it is possible. Put
set linesize 32767
set trimout on
before your select.
Enjoy.
I have some issues with how sqlplus output is formatted for the terminal and I am just thinking of writing a script around sqlplus and fixing these.
On the other hand, wow that seems really lame. Because Oracle has several tons of tools written. Yet it seems difficult to get what I want. Does anyone have another suggestion?
First, I want smarter column widths. If I create a table with a column whose max size is 200 characters but then I put "abc", "xyz" and "123" in it, do I need a 200-space wide column on the terminal to display the contents? I do not think so. I think I need 3 characters plus a couple for padding. Yet Oracle insists on giving me a 200-character wide column. Unless there is somewhere to fix this.
Second, I want easy access to a sideways display of the columns, like using \G at the end of the command in MySQL. I know there is a way to do this in Oracle but it seemed complicated. Why could there not just be a simple switch? Like a \G at the end of the command? There can be if I wrap the output to sqlplus and do this myself.
So, the question seems to be this. Should I write a simple script around sqlplus to give me what I want, or is there a way to get Oracle to give me this behavior In sqlplus? And if there is, how much extra information will I have to stuff into my head to make this work? Because it does not seem as though it should be very complicated. But Oracle is certainly not making it easy.
First of all I suggest you look over the SQL*plus reference - you might find some useful tips there like adjusting a column width
COL column_name for a20
you can set up your own settings in the GLOGIN file. over time, like any other CMD, you'll get your preferences just right.
To describe a table you can use DESC. if you want more data write your own script and reuse it with #.
If all this doesn't work for you, you can always switch to a GUI like Toad or SQL developer.
EDIT:
I'm adding one of my own scripts to show you some tricks on how to make SQL*Plus more friendly on the command line. This one is for getting segment sizes.
/* This is the trick - clears &1 and &2 if received an empty string */
set ver off feed off
col 1 new_v 1
col 2 new_v 2
select 1,2 from dual where 1=0;
variable p_owner varchar2(30)
variable p_segment varchar2(30)
/* set bind variables */
begin
:p_owner := '&1';
:p_segment := '&2';
end;
/
set feed 1
break on segment_type skip 1
column MB for a25
select
segment_type,
decode(gi_segment_name + gi_segment_type + gi_tablespace_name , 3 ,'...Grand Total', segment_name) SEGMENT_NAME,
to_char(round(MB,3),'99,999,999.99') MB ,
nvl(tablespace_name,'-*-') tablespace_name
from (
select tablespace_name , segment_type , segment_name , sum(bytes/1024/1024) MB ,
grouping_id(segment_name) gi_segment_name ,
grouping_id(segment_type) gi_segment_type ,
grouping_id(segment_type) gi_tablespace_name
from dba_segments
where ((:p_owner is null and owner = user) or owner like upper(:p_owner))
and (:p_segment is null or segment_name like upper('%'||:p_segment||'%'))
group by rollup(tablespace_name, segment_type , segment_name)
)
where not (gi_segment_name = 1 and gi_segment_type = 0 and gi_tablespace_name = 0)
order by decode(segment_type,'TABLE','0','TABLE PARTITION','1','INDEX','2','INDEX PARTITION','3',segment_type) ,
(case when segment_name like '.%' then 'z' else 'a' end) ,
gi_segment_name ,
MB desc ,
segment_name;
clear break
/* clear definition for &1 and &2 after being used.
allows the variable to be null the next run. */
undefine 1
undefine 2
I'll walk you through some of the things iv'e done here
The script accepts two parameters. The first 4 lines clears the
parameter if none received. if you don't do this SQL*Plus will prompt
you for them. And we dont want that.
Setting the binds was more of a big deal in past version. It's
intended to save Hard / Soft parse. latest version solve this
problem. It's still a best practice though.
The break is a nice touch. You'll see it.
The grouping Id show me the sub totals on several levels.
I've added two parameter, owner and segment name. both can contain
wild card. both can be null. If non provided the query will fetch the
current user segments.
Order by decode enabled me to set a custom sort order for different
segment types. You can change it as you wish.
I'm executing the script like this
my segments :
#seg
Scott's segments
#seg scott
Scott's Emp related segments
#seg scott emp
I have similar scripts for session, longops, wait events, tables, constraints, locks, kill session etc .... during my daily routine i rarely write SQL for querying this stuff any more.
I have a script that i use to load my data into my tables in Oracle (by a list of insert statements). How can i get the execution time of the entire loading process? I have tried with set timing on, but that gives me a duration for each insert statement and not the entire process. The script is shown below:
spo load.log
prompt '**** load data ****'
set termout off
##inserts.sql
commit;
set termout on
prompt '**** done ****'
spo off
exit;
Not sure why everybody is making it so complex. Simple as:
SQL> set timing on
SQL> select 1 from dual;
1
----------
1
1 row selected.
Elapsed: 00:00:00.00
SQL>
It old question but i have found easy way to measure time of running a script in sqlplus. You just have to add this on the beginning
timing start timing_name
And this on the end of a script
timing stop
More information about this command can be found at Oracle's SQL*Plus® User's Guide and Reference: Collecting Timing Statistics
Try this, add the following at the beginning and it remembers the current time:
set serveroutput on
variable n number
exec :n := dbms_utility.get_time
Add this at the end and it calculates the time elapsed:
exec :n := (dbms_utility.get_time - :n)/100
exec dbms_output.put_line(:n)
If you are on Unix, you can also do it like that:
time sqlplus #inserts.sql
It will print:
real 0m9.34s
user 0m2.03s
sys 0m1.02s
The first line gives the total execution time.
What you're describing is essentially a way to audit the script's execution. Whether it's an elapsed time, or specific start and end times you're capturing you want to log them properly to see if things went well (or if not, why not).
Here's a template similar to what we use for capturing and logging all database activity we are implementing. We use it via sqlplus.exe for all DDL updates (e.g. CREATE TABLE) and for inserts into setup tables.
--Beginning of all SQL scripts:
set serveroutput on feedback on echo on verify on sqlblanklines on timing on define on
col time new_v v_time
col name new_v v_name
col user new_v v_user
select name, USER, to_char(sysdate, 'YYYYMMDD-HH24MISS') time from v$database;
--Creates a new log file every time the script is run, and it's immediately
--obvious when it was run, which environment it ran in, and who ran it.
spool &v_time._&v_name._&v_user..log
--Run the select again so it appears in the log file itself
select name, USER, to_char(sysdate, 'YYYYMMDD-HH24MISS') time from v$database;
Place the body of your work here.
--End of all SQL scripts:
select name, USER, to_char(sysdate, 'YYYYMMDD-HH24MISS') time from v$database;
spool off