Remove/Prevent First Blank from Oracle CSV MARKUP output file - windows

I am currently writing scripts to output table data to CSV files so that they can be ingested into another platform. I'd much rather use the set markup CSV on quote on; SQL option to generate my output file. as it makes the script much cleaner and a whole lot simpler to write.
The problem, however, is that no matter how hard I try, I cannot seem to find an option that will prevent the output file from starting with a blank line. I need the column headers in the very first line and not in the second line.
The scripts need to run on a Windows Server against an Oracle 12c (12.1.0.2.0) Database, using SQLPLUS to execute the SQL commands and PowerShell/batch as the scripting wrapper.
After searching for some time and trying various options that only seem to have any effect on older approaches to CSV outputs. Eventually, I found the Oracle documentation for the SET command and discovered that things like SET NEWPAGE, SET TRIMSPOOL have no effect when using SET MARK[UP].
Oracle SET Command documentation is here: https://docs.oracle.com/en/database/oracle/oracle-database/12.2/sqpug/SET-system-variable-summary.html#GUID-0AA910C4-C22A-4A9E-BE13-AAA059CC7919
Is there a simple way to get rid of or suppress the first line that is blank? I really don't want to have to write further file processing scripts that go and remove the line from the file using PowerShell/Batch. That just seems like a really dumb thing to have to do...
A simplified version of my SQL is below and the output still includes the blank first line.
column thespoolfile new_value thespoolfile
select 'D:\Temp\outfile-detail-' || TO_CHAR( sysdate, 'YYYYMMDDHH24MI' ) || '.csv' as thespoolfile from dual;
set markup csv on quote on;
SET FEEDBACK OFF;
set trimspool on;
SET HEADSEP off;
SET SQLBLANKLINES off;
SET termout OFF;
spool &thespoolfile;
set define off;
select *
from <table>;
spool off;
exit


You need to use the following option
SET NEWPAGE NONE
It will suppress the first blank line, which is bothering you.
Cheers!!

Related

Unable to export dynamic Oracle table to properly formatted CSV using sqlplus

I am trying to substitute a Python/pandas export process and go direct from Oracle to csv. I have seen a couple of posts such as this one here
Suppose I have a table in Oracle with three columns: ColA, ColB, ColC. I want to employ a command line utility that takes as input an SQL command and generates a CSV file that would look like any standard CSV file with a header line and rows of values:
ColA,ColB,ColC
A,B,C
D,E,F
...
In SQL Server, there is a utility called bcp I can use which looks like this:
bcp "SELECT 'ColA', 'ColB', 'ColC' UNION ALL Select ColA, ColB, ColC From
myTable" queryout "C:\Temp\csvfiles\output.csv" -c -CRAW -S localhost -u
my_userid -p my_password -d my_database
Easy-peasy. But in looking for the Oracle equivalent, not so much.
I took a look here but the author wants you to create a UTL_FILE. That doesn't seem like a possibility since this is supposed to be a completely generic process that gets built and run from Python. I took a look at sqlplus but I can't seem to come up with a way to create a typical CSV similar to bcp. The docs here weren't terribly clear on this either. It's such a common format I expected to see tons of examples.
Sites like this here show how to output a fixed-length CSV using sqlplus but don't show you how to add a header row and values are padded, which might be OK, but not desirable. This was about the best I could come up with:
set colsep ,
set headsep off
set pagesize 0
set trimspool on
set linesize 200
set numwidth 50
SET TERMOUT OFF
Spool output.csv
SELECT ColA,
ColB,
ColC
FROM MyTable;
Spool off
exit
So, is there an equivalent tool/command in Oracle to the trusty bcp, or is there a sample somewhere using sqlplus that can generate a csv similar to the one above with a header line and without all the leading spaces? Or is there another option to generate dynamic CSV files from random Oracle tables? Or am I stuck using Python, pyodbc and pandas to get my nicely-formatted CSV files? TIA

Oracle's SQLcl utility(commandline executable sql or sql.exe) could help you achieve it.
Here's the download link SQLcl. It's free.
In order to export a file in the CSV format, you may simply specify
SET SQLFORMAT CSV
If you want a different delimiter, use
SET SQLFORMAT DELIMITED '|'
Here's a sample output with the header
SQL> select * from employees;
"EMPLOYEE_ID","FIRST_NAME","LAST_NAME","EMAIL","PHONE_NUMBER","HIRE_DATE","JOB_ID","SALARY","COMMISSION_PCT","MANAGER_ID","DEPARTMENT_ID"
198,"Donald","OConnell","DOCONNEL","650.507.9833",21-06-07,"SH_CLERK",2600,,124,50
199,"Douglas","Grant","DGRANT","650.507.9844",13-01-08,"SH_CLERK",2600,,124,50
200,"Jennifer","Whalen","JWHALEN","515.123.4444",17-09-03,"AD_ASST",4400,,101,10
201,"Michael","Hartstein","MHARTSTE","515.123.5555",17-02-04,"MK_MAN",13000,,100,20
202,"Pat","Fay","PFAY","603.123.6666",17-08-05,"MK_REP",6000,,201,20
203,"Susan","Mavris","SMAVRIS","515.123.7777",07-06-02,"HR_REP",6500,,101,40
204,"Hermann","Baer","HBAER","515.123.8888",07-06-02,"PR_REP",10000,,101,70
205,"Shelley","Higgins","SHIGGINS","515.123.8080",07-06-02,"AC_MGR",12008,,101,110
206,"William","Gietz","WGIETZ","515.123.8181",07-06-02,"AC_ACCOUNT",8300,,205,110
100,"Steven","King","SKING","515.123.4567",17-06-03,"AD_PRES",24000,,,90
101,"Neena","Kochhar","NKOCHHAR","515.123.4568",21-09-05,"AD_VP",17000,,100,90
102,"Lex","De Haan","LDEHAAN","515.123.4569",13-01-01,"AD_VP",17000,,100,90
103,"Alexander","Hunold","AHUNOLD","590.423.4567",03-01-06,"IT_PROG",9000,,102,60
104,"Bruce","Ernst","BERNST","590.423.4568",21-05-07,"IT_PROG",6000,,103,60
105,"David","Austin","DAUSTIN","590.423.4569",25-06-05,"IT_PROG",4800,,103,60
106,"Valli","Pataballa","VPATABAL","590.423.4560",05-02-06,"IT_PROG",4800,,103,60
107,"Diana","Lorentz","DLORENTZ","590.423.5567",07-02-07,"IT_PROG",4200,,103,60
108,"Nancy","Greenberg","NGREENBE","515.124.4569",17-08-02,"FI_MGR",12008,,101,100
109,"Daniel","Faviet","DFAVIET","515.124.4169",16-08-02,"FI_ACCOUNT",9000,,108,100
Additionally, all the SQL* Plus commands are supported with few more specific to SQLcl. See this command line reference

SET MARKUP CSV ON - use this in sqlplus before executing the query to get the data in csv format

Spooling to a file with a name containing a space and script's parameter in sqlplus?

I'm trying to change a script with a parameter to spool the output into specific path. After searching Stack Overflow I came up with:
column filename new_val filename
select '"i:\Direct bank\incoming\ROSE\report_zgod_' || '^1' ||'.csv"' filename from dual;
spool &filename
This creates correct filename "i:\Direct bank\incoming\ROSE\report_zgod_2017-08-28.csv" (I have tried to put it literally instead of &filename and the file has been created), but it still does not write a file. My guess is that's because there is a space in the path.
I have also tried
spool "&filename"
spool '&filename'
but to no avail.
I can't avoid parameter (it's used in other places in script and it can't be really calculated from SYSDATE) and I can't change the path (it has to be somewhere under "I:\Direct bank").
How can I have both the parameter and the space in spool file's name?

OK, I found the answer in comments to https://asktom.oracle.com/pls/apex/f?p=100:11:0::::P11_QUESTION_ID:3581757800346555562 .
The author of the original script has changed the defaults, so that I have to use
spool ^filename
instead of
spool &filename
In case you have similar problem, the orginal code contained:
set define "^"
set sqlprefix "^"
And that was the reason I could not get the file created.

When the path has a space in it then try using the short form of the path name. So, to SPOOL to /opt/oracle/oradata/Custom Scripts/orders_between_dates.txt; it's...
SQL> SPOOL /opt/oracle/oradata/Custom~1/orders_between_dates.txt;
If you're running it from a .sql script then put "" around it, like this...
SET FEEDBACK OFF;
SET MARKUP CSV ON;
SET VERIFY OFF;
SPOOL "/opt/oracle/oradata/Custom~1/orders_between_dates.txt"
SELECT * FROM orders_detail WHERE order_date BETWEEN '&1' AND '&2';
SPOOL OFF;
EXIT;
The command to run this script for a rolling seven day window would be something like...
> sqlplus ot/Orcl1234#xepdb1 #"/opt/oracle/oradata/Custom Scripts/orders_between_dates.sql" $(date +%d-%b-%Y -d '-7 days') $(date +%d-%b-%Y)

How to spool file1 or file2 from a single SQL file based on the input?

I have a SQL file like the following:
SET ECHO OFF
set feed off
set verify off
set head off
set pagesize 0
set space 0
set trimspool on
set line 250
spool subir.rpt
select 'Subir' from dual;
spool off
SET ECHO OFF
set feed off
set verify off
set head off
set pagesize 0
set space 0
set trimspool on
set line 250
spool vishal.rpt
select 'vishal' from dual;
spool off
Let's call this file spool.sql. I need to enable creation of spool file based on the input spool.sql file receives. For example, if it receives "subir.rpt" The first portion of the SQL file should run and subir.rpt should be produced; if it is vishal.rpt, the second portion should run and vishal.rpt should get generated.
How can this be achieved?
Edit: This SQL file is getting called from a shell script. The shellscript is like the following:
sqlplus -s dbread$HOST_CONNECT_STR/dbread<<endplus
#spool.sql
endplus

What is launching this spool.sql? A shell script? Database function?
If it's a shell script, I would either use a parameter to this script or split it into two separate .sql files and call the appropriate one.
You could also save the file value into a temporary table prior to calling this script and select it before spooling the file.
There are a lot of options but it all depends on how you call this script and how extensible you want it to be.

The simplest way would be to use the supplied parameter dynamically as the spoolfile:
set echo off
set feed off
set verify off
set head off
set pagesize 0
set space 0
set trimspool on
set line 250
spool &1
spool
select 'Subir' from dual where '&1' = 'Subir.rpt';
select 'vishal' from dual where '&1' = 'vishal.rpt';
spool off
(The spool command on its own is just to print the current spoolfile name for demo purposes.)

remove blank lines from csv using shell script for oracle

Hi am using following shell script to create csv from oracle database
export ORACLE_HOME=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server
export PATH=$PATH:$ORACLE_HOME/bin
sqlplus -s user1/user1<<!
Spool on
set heading off
SET FEEDBACK OFF
set wrap off
set colsep ,
SET LINESIZE 200
set trims on
Spool /home/ist/username.csv
select 'username','userid' from dual;
select username,userid from users;
Spool off
EXIT
!
I am getting following output
as you can see there is blank line at first and third row
but am expecting file without blank lines.
Any help will be appreciated.

Use the
SET PAGESIZE 0
command to avoid the blank lines. this also suppresses column headings, so you can remove
SET HEADING OFF
The command
SPOOL on
does not make sense because it starts spooling in a file named on.lst. So remove this command, too.
If you want to display the heading with the column name
you can try the following settings
set HEADING ON
SET FEEDBACK OFF
set WRAP OFF
set COLSEP ,
SET LINESIZE 32767
set NEWPAGE none
set UNDERLINE OFF
set TRIMSPOOL ON
set TRIMOUT ON
set PAGESIZE 50000
´heading on´ is the default so you must not set it. It enables the display of the column names when a select starts. underline off suppresses the '---' line between column names and data of a select. pages 50000 sets the pagesize to its maximum value (Oracle 11.2). linesize 32767 sets the linesize to its maximum value (Oracle 11.2). newpage none is necessary to suppress this empty line at the beginning of a page that was the primary concern of your posting.
All this can be found in the SQL*Plus Command Reference
The termout off parameter suppresses only output created by a scripts that is executed with the # or ## command. It dos not suppress out by a command entered in the SQL*plus console. If you use
sqlplus user/passw#connect <<!
...
!
you use the here-document syntax of the shell language which simulates the interactive input. So put your sql commands in a script, e.g. script.sql, and execute
sqlplus user/passw#connect #script.sql
then termout off will suppress terminal output.
Instead of
colsep ,
select username,userid
...
which returns something like
user1 , 14
nextuser , 236
myuser , 11
...
you can use leave the COLSEP unchanged and execute
select username||','||userid
...
to get the following output
user1,14
nextuser,236
myuser,11
...
Maybe this is useful
https://dba.stackexchange.com/a/64620/2047

set trimspool on
this will work for you

put this on your script
sed -i '/^\s*$/d'/home/ist/username.csv

Remove lines in /home/ist/username.csv without a , using
grep "," /home/ist/username.csv > /home/ist/username2.csv
# or
sed -i '/^[^,]*$/ d' /home/ist/username.csv

SQLPLUS 12.2 has an feature to output CSV, but you need to configure some settings to remove blank lines on it. Here is my approach
SET HEADING OFF;
SET PAGES 0;
SET SQLBLANKLINES ON;
set echo off;
set autotrace off;
set feedback off;
SET TRIM ON;
SET MARKUP CSV ON DELIMITER | QUOTE OFF;

Export to a csv file

I try many times to export my result into .csv file but I'm always fail. I read lot of articles related in my problem.
Look at my query:
spool sample.csv
SELECT /*html*/(CODE_SALESROOM) POS_ID, (NAME_SALESROOM) POS_NAME
FROM OWNER_DWH.DC_SALESROOM
WHERE CODE_SALESROOM NOT IN ('XAP', 'XNA', '10001');
spool off;
The output of this, when I export in a csv file is only the query not the real result.
What's wrong in my command to export the result?
Please help me. Thanks

Make sure that your query returns values. You can set various parameters of Sql plus before spooling to get desired out put. Below is the list that I use before spooling. And since your are generating csv, it should be comma separated
set echo off
set termout off
set define off
set heading off
set pagesize 10000
set linesize 500
set feedback off
set verify off
set trimspool on
spool sample.csv
SELECT 'POS_ID,POS_NAME' from dual; -- header list
SELECT CODE_SALESROOM || ',' || NAME_SALESROOM
FROM OWNER_DWH.DC_SALESROOM
WHERE CODE_SALESROOM NOT IN ('XAP', 'XNA', '10001');
spool off;
You can modify the set parameters based on your requirement.

Resources