I am required to spool a csv from a table in Oracle, using sqlplus. Following is the format required:
"HOST_SITE_TX_ID","SITE_ID","SITETX_TX_ID","SITETX_HELP_ID"
"664436565","16","2195301","0"
"664700792","52","1099970","0"
Following is the relevant piece of the shell script I wrote:
sqlplus -s $sql_user/$sql_password#$sid << eof >> /dev/null
set feedback off
set term off
set linesize 1500
set pagesize 11000
--set colsep ,
--set colsep '","'
set trimspool on
set underline off
set heading on
--set headsep $
set newpage none
spool "$folder$filename$ext"
select '"'||PCL_CARRIER_NAME||'","'||SITETX_EQUIP_ID||'","'||SITETX_SITE_STAT||'","'||SITETX_CREATE_DATE||'","'||ADVTX_VEH_WT||'"'
from cvo_admin.MISSING_HOST_SITE_TX_IDS;
spool off
(I have used some commented statements in, to signify the things that I tried but couldn't get to work)
The output I receive is:
'"'||PCL_CARRIER_NAME||'","'||SITETX_EQUIP_ID||'","'||SITETX_SITE_STAT||'","'||SITETX_CREATE_DATE||'","'||ADVTX_VEH_WT||'"'
"TRANSPORT INC","113","00000000","25-JAN-13 10.17.51 AM",""
"TRANSPORT INC","1905","00000000","25-JAN-13 05.06.44 PM","0"
Which shows that the header is messed up - it is literally printing the whole string that should have been interpreted as an sql statement, as is the case with the data displayed.
Options I am considering:
1) Using colsep
set colsep '","'
spool
select * from TABLE
spool off
This introduces other problems as the data having leading and trailing spaces, first and the last values in the files are not enclosed by quotes
HOST_SITE_TX_ID"," SITE_ID"
" 12345"," 16"
" 12345"," 21
I concluded that this method gives me more heartburn than the one I described earlier.
2) Getting the file and use a regex to modify the header.
3) Leaving the header altogether and manually adding a header string at the beginning of the file, using a script
Option 2 is more doable, but I was still interested in asking, if there might be a better way to format the header somehow, so it comes in a regular csv, (comma delimited, double quote bounded) format.
I am looking to do as less hard coding as possible - the table I am exporting has around 40 columns and I am currently running the script for around 4 million records - breaking them in a batch of around 10K each. I would really appreciate any suggestions, even totally different from my approach - I am a programmer in learning.
One easy way to have a csv with just one header is to do
set embedded on
set pagesize 0
set colsep '|'
set echo off
set feedback off
set linesize 1000
set trimspool on
set headsep off
the embedded is a hidden option but it is important to have JUST one header
This is how I created a header:
set heading off
/* header */
SELECT '"'||PCL_CARRIER_NAME||'","'||SITETX_EQUIP_ID||'","'||SITETX_SITE_STAT||'","'||SITETX_CREATE_DATE||'","'||ADVTX_VEH_WT||'"'
FROM
(
SELECT 'PCL_CARRIER_NAME' AS PCL_CARRIER_NAME
, 'SITETX_EQUIP_ID' AS SITETX_EQUIP_ID
, 'SITETX_SITE_STAT' AS SITETX_SITE_STAT
, 'SITETX_CREATE_DATE' AS SITETX_CREATE_DATE
, 'ADVTX_VEH_WT' AS ADVTX_VEH_WT
FROM DUAL
)
UNION ALL
SELECT '"'||PCL_CARRIER_NAME||'","'||SITETX_EQUIP_ID||'","'||SITETX_SITE_STAT||'","'||SITETX_CREATE_DATE||'","'||ADVTX_VEH_WT||'"'
FROM
(
/* first row */
SELECT to_char(123) AS PCL_CARRIER_NAME
, to_char(sysdate, 'yyyy-mm-dd') AS SITETX_EQUIP_ID
, 'value3' AS SITETX_SITE_STAT
, 'value4' AS SITETX_CREATE_DATE
, 'value5' AS ADVTX_VEH_WT
FROM DUAL
UNION ALL
/* second row */
SELECT to_char(456) AS PCL_CARRIER_NAME
, to_char(sysdate-1, 'yyyy-mm-dd') AS SITETX_EQUIP_ID
, 'value3' AS SITETX_SITE_STAT
, 'value4' AS SITETX_CREATE_DATE
, 'value5' AS ADVTX_VEH_WT
FROM DUAL
) MISSING_HOST_SITE_TX_IDS;
This is how you add a pipe delimited header to SQL statements. Once you spool it out that "something" wont be there
-- this creates the header
select 'header_column1|header_column2|header_column3' as something
From dual
Union all
-- this is where you run the actual sql statement with pipes in it
select
rev.value1 ||'|'||
rev.value2 ||'|'||
'related_Rel' as something
from
...
In Oracle 19 you can use set markup csv on to ensure that csv outputs are created.
You can also set the delimiter and optional quote or even spool html if you prefer
You can read more here
set markup csv on
spool "$folder$filename$ext"
select q'|wow, I can't beleive he said "hello, how are you?", can you beleive it!|' as text
from dual;
spool off
quit;
Related
currently i use below code to spool csv file but i have find some line length more than 4000 characters , so it will no propey display
set echo off
set feedback off
set heading on
set linesize 32000
set pagesize 0
set termout off
set trim off
set trimspool on
set array 100
set underline off
set wrap off
set flush off
set verify off
set embedded on
select cloumnA||','||ColumnB ||','||ColumnC ..... from my_tab_name ；
so i modify my code but the result is not my want.
set echo off
set feedback off
set heading on
set linesize 32000
set pagesize 0
set termout off
set trim off
set trimspool on
set array 100
set underline off
set wrap off
set flush off
set verify off
set embedded on
select cloumnA,ColumnB ,ColumnC ..... from my_tab_name ；
the result 1:
A_value,B_value,C_value..... (the result is correct ,but only can show 4000 characters)
the result 2:
A_value
B_value
C_value
....
Anyone can help,thanks a lot!
When you concatenate columns (cloumnA||','||ColumnB ||','||ColumnC), the result is a VARCHAR2, which can't be more than 4000 characters (in typical circumstances).
See this similar question for some different options.
You could use COLSEP:
set colsep ','
spool myfile.csv
select cloumnA,ColumnB,ColumnC ..... from my_tab_name;
Which will get you something like
A_value ,B_value ,C_value
A_value ,B_value ,C_value
A_value ,B_value ,C_value
Which most people don't like, but it doesn't have a 4000-character line limit.
If you're using SQL Developer, you can use the easy /*csv*/ hint
select /*csv*/ cloumnA,ColumnB,ColumnC ..... from my_tab_name;
And if it's possible for you to use the newer SQLcl instead of SQL*Plus, you can use set sqlformat:
set sqlformat csv
spool myfile.csv
select cloumnA,ColumnB,ColumnC ..... from my_tab_name;
In Oracle SQL Developer, I can get simple query results returned in the 'Query Results' grid, but if I need to use variable in script, I need to use the 'Run Script' option and my results show up in 'Script Output' window, and I can't export it to csv format. Here is my sample code:
var CatCode char(5) ;
exec :CatCode := 'ZK';
SELECT * FROM Products WHERE CategoryCode = :CatCode;
Any help would be appreciated.
Thanks.
Just add a /*csv*/ to your query, the tool will bring back the output in CSV automatically when executed as a script (F5).
Or use a substitution variable instead. &Var vs :Var, run with F9, SQLDev will prompt you for the value.
VAR stcode CHAR(2);
EXEC :stcode := 'NC';
SELECT /*csv*/
*
FROM
untappd
WHERE
venue_state =:stcode;
Or to go straight to the grid so you can use can use the Grid Export feature.
SELECT
*
FROM
untappd
WHERE
venue_state =:stcode2;
Execute with Ctrl+Enter or F9
Supply the input parameter in the pop up dialog, click OK.
Shazaam.
Here you go you can run this one to be ensure. it's running.
set colsep , -- separate columns with a comma
set pagesize 0 -- No header rows
set trimspool on -- remove trailing blanks
set headsep off -- this may or may not be useful...depends on your headings.
set linesize X -- X should be the sum of the column widths
set numw X -- X should be the length you want for numbers (avoid scientific notation on IDs)
spool C:\Users\**direcotory**\sql\Test1.csv; --this is file path to save data
var CatCode char(5) ;
exec :CatCode := 'ZK';
SELECT * FROM Products WHERE CategoryCode = :CatCode;
spool off;
Thanks #thatjeffsmith and Paras, spool option gave me new direction and it worked. I slightly changed your code and it works great.
var CatCode char(5) ;
exec :CatCode := 'ZK';
set feedback off;
SET SQLFORMAT csv;
spool "c:\temp\spoolTest.csv"
SELECT * FROM Products WHERE CategoryCode = :CatCode;
spool off;
SET SQLFORMAT;
set feedback on;
I have the following in Unix under my shell script:
SET linesize 2000
SET pagesize 50000
SET ECHO OFF
SET FEEDBACK OFF
SET VERIFY OFF
SET MARKUP HTML OFF
set numwidth 17
set colsep ","
select accountnum, cardnum, name, address, number from employee;
Once it generates the output in .csv, cardnum and accountnum becomes scientific like:
5.30706E+15
I want it to be:
501889176278289
I know how to change it in Excel but since I am sending these reports to a client. I want it to go to them in the correct format so they don't have to change anything.
I would suggest this:
SELECT TO_CHAR(accountnum,'9999999999999999999') accountnum
, TO_CHAR(cardnum,'9999999999999999999') cardnum
, name, address, number
FROM employee;
Except of TO_CHAR suggested in other answer you can try:
set numformat 9999999999999999999 that should solve your problem.
Another way is COLUMN accountnum FORMAT 9999999999999999999
I have a shell with one command line opening sql plus and calling a sql file
$ORACLE_HOME/bin/sqlplus id/psw #SQL_DIRECTORY/myfile.sql
in this file I write this :
spool myfile.csv
select 'column 01', ';', 'column 02' from dual;
select myColumn1, ';', mycolum2 from mytable
spool off
It works good BUT on the first line of the the first column of each select there is a special character.
Why is that? how can I get rid of this
Thanks
From the description it sounds like you have a login.sql or glogin.sql user profile that is issuing set newpage 0:
Sets the number of blank lines to be printed from the top of each page to the top title. A value of zero places a formfeed at the beginning of each page (including the first page) and clears the screen on most terminals. If you set NEWPAGE to NONE, SQL*Plus does not print a blank line or formfeed between the report pages.
In Excel that does show up as an unprintable character with a question mark in a small square; in Notepad++ it seems to show as FF in reverse colouring; and in some editors (e.g. Vim) it shows as ^L. This is the ASCII form feed character, decimal 12 or 0xC.
You can either reset that with set newpage none, or make it irrelevant with set pagesize 0, which has the convenient side effect of removing the column headers and separators. You may also want to set feedback off if you aren't already.
Samples with contrasting settings:
set newpage 0;
set feedback on;
set pagesize 100;
spool ctrl.csv
select 'head 1;head 2' from dual;
select sysdate, ';', systimestamp from dual;
spool off
^L'HEAD1;HEAD2'
-------------
head 1;head 2
1 row selected.
^LSYSDATE ' SYSTIMESTAMP
------------------- - ---------------------------------------------------------------------------
2015-09-16 15:45:42 ; 16-SEP-15 15.45.42.333627 +01:00
1 row selected.
And
set newpage none;
set feedback off;
set pagesize 0;
spool ctrl.csv
select 'head 1;head 2' from dual;
select sysdate, ';', systimestamp from dual;
spool off
head 1;head 2
2015-09-16 15:46:11 ; 16-SEP-15 15.46.11.274863 +01:00
I have a query that returns a lot of data into a CSV file. So much, in fact, that Excel can't open it - there are too many rows. Is there a way to control spool to spool to a new file everytime 65000 rows have been processed? Ideally, I'd like to have my output in files named in sequence, such as large_data_1.csv, large_data_2.csv, large_data_3.csv, etc...
I could use dbms_output in a PL/SQL block to control how many rows are output, but then how would I switch files, as spool does not seem to be accessible from PL/SQL blocks?
(Oracle 10g)
UPDATE:
I don't have access to the server, so writing files to the server would probably not work.
UPDATE 2:
Some of the fields contain free-form text, including linebreaks, so counting line breaks AFTER the file is written is not as easy as counting records WHILE the data is being returned...
Got a solution, don't know why I didn't think of this sooner...
The basic idea is that the master sqplplus script generates an intermediate script that will split the output to multiple files. Executing the intermediate script will execute multiple queries with different ranges imposed on rownum, and spool to a different file for each query.
set termout off
set serveroutput on
set echo off
set feedback off
variable v_rowCount number;
spool intermediate_file.sql
declare
i number := 0;
v_fileNum number := 1;
v_range_start number := 1;
v_range_end number := 1;
k_max_rows constant number := 65536;
begin
dbms_output.enable(10000);
select count(*)
into :v_err_count
from ...
/* You don't need to see the details of the query... */
while i <= :v_err_count loop
v_range_start := i+1;
if v_range_start <= :v_err_count then
i := i+k_max_rows;
v_range_end := i;
dbms_output.put_line('set colsep ,
set pagesize 0
set trimspool on
set headsep off
set feedback off
set echo off
set termout off
set linesize 4000
spool large_data_file_'||v_fileNum||'.csv
select data_string
from (select rownum rn, data_object
from
/* Details of query omitted */
)
where rn >= '||v_range_start||' and rn <= '||v_range_end||';
spool off');
v_fileNum := v_fileNum +1;
end if;
end loop;
end;
/
spool off
prompt executing intermediate file
#intermediate_file.sql;
set serveroutput off
Try this for a pure SQL*Plus solution...
set pagesize 0
set trimspool on
set headsep off
set feedback off
set echo off
set verify off
set timing off
set linesize 4000
DEFINE rows_per_file = 50
-- Create an sql file that will create the individual result files
SET DEFINE OFF
SPOOL c:\temp\generate_one.sql
PROMPT COLUMN which_dynamic NEW_VALUE dynamic_filename
PROMPT
PROMPT SELECT 'c:\temp\run_#'||TO_CHAR( &1, 'fm000' )||'_result.txt' which_dynamic FROM dual
PROMPT /
PROMPT SPOOL &dynamic_filename
PROMPT SELECT *
PROMPT FROM ( SELECT a.*, rownum rnum
PROMPT FROM ( SELECT object_id FROM all_objects ORDER BY object_id ) a
PROMPT WHERE rownum <= ( &2 * 50 ) )
PROMPT WHERE rnum >= ( ( &3 - 1 ) * 50 ) + 1
PROMPT /
PROMPT SPOOL OFF
SPOOL OFF
SET DEFINE &
-- Define variable to hold number of rows
-- returned by the query
COLUMN num_rows NEW_VALUE v_num_rows
-- Find out how many rows there are to be
SELECT COUNT(*) num_rows
FROM ( SELECT LEVEL num_files FROM dual CONNECT BY LEVEL <= 120 );
-- Create a master file with the correct number of sql files
SPOOL c:\temp\run_all.sql
SELECT '#c:\temp\generate_one.sql '||TO_CHAR( num_files )
||' '||TO_CHAR( num_files )
||' '||TO_CHAR( num_files ) file_name
FROM ( SELECT LEVEL num_files
FROM dual
CONNECT BY LEVEL <= CEIL( &v_num_rows / &rows_per_file ) )
/
SPOOL OFF
-- Now run them all
#c:\temp\run_all.sql
Use split on the resulting file.
utl_file is the package you are looking for. You can write a cursor and loop over the rows (writing them out) and when mod(num_rows_written,num_per_file) == 0 it's time to start a new file. It works fine within PL/SQL blocks.
Here's the reference for utl_file:
http://www.adp-gmbh.ch/ora/plsql/utl_file.html
NOTE:
I'm assuming here, that it's ok to write the files out to the server.
Have you looked at setting up an external data connection in Excel (assuming that the CSV files are only being produced for use in Excel)? You could define an Oracle view that limits the rows returned and also add some parameters in the query to allow the user to further limit the result set. (I've never understood what someone does with 64K rows in Excel anyway).
I feel that this is somewhat of a hack, but you could also use UTL_MAIL and generate attachments to email to your user(s). There's a 32K size limit to the attachments, so you'd have to keep track of the size in the cursor loop and start a new attachment on this basis.
While your question asks how to break the greate volume of data into chunks Excel can handle, I would ask if there is any part of the Excel operation that can be moved into SQL (PL/SQL?) that can reduce the volume of data. Ultimately it has to be reduced to be made meaningful to anyone. The database is a great engine to do that work on.
When you have reduced the data to more presentable volumes or even final results, dump it for Excel to make the final presentation.
This is not the answer you were looking for but I think it is always good to ask if you are using the right tool when it is getting difficult to get the job done.