Spool CSV file using BASH Script | Errors with Output in generated CSV - oracle

I need to spool the output of tables from db to CSV file.
I checked and referred to How do I spool to a CSV formatted file using SQLPLUS?
But it's not working successfully. What I am getting is only the first 3 columns being separated by , and not the remaining. Others are getting separated by newline.
EDIT : Schema Details of Table being Spooled.
WWID VARCHAR2 (5 Byte)
TIMELOG_DATE DATE
TOTAL_HOURS NUMBER (10,2)
ACTIVITY VARCHAR2 (100 Byte)
SUBACTIVITY VARCHAR2 (100 Byte)
REF_PROJECT_ID VARCHAR2 (30 Byte)
REF_PROJECT_DESC VARCHAR2 (250 Byte)
WORK_REQUEST_ID VARCHAR2 (30 Byte)
EMP_COMMENTS VARCHAR2 (100 Byte)
APPROVER_COMMENTS VARCHAR2 (100 Byte)
Script:
echo "\n>>> ******Data Processing Started at `date '+%d-%m-%Y %T %Z'`******" >>${LOGFILE}
sqlplus $fims_user/$fims_pwd << EOF
set serveroutput on;
set colsep , ;-- separate columns with a comma
set pagesize 0 ;-- No header rows
set trimspool on ;-- remove trailing blanks
set headsep off ;-- this may be useful...depends on your headings.
spool /home/fimsctl/datafiles/outbound/timelog/timelog_file_`date +%y%m%d`.csv
select * from FIMS_OWNER.TIMELOG_EXTRACT_OUTBOUND_T;
commit;
spool off
exit
EOF
echo "\n>>> ******Data Load Completed at `date '+%d-%m-%Y %T %Z'`******" >>${LOGFILE}
echo "End of the script">> ${LOGFILE}
And Output in CSV i am getting is:
SQL> select * from FIMS_OWNER.TIMELOG_EXTRACT_OUTBOUND_T;
iv315,29-DEC-14, 8
DUMMY
REF01
New Ref Project of type CPRM
66
NA
iv315,30-DEC-14, 8
DUMMY
REF01
New Ref Project of type CPRM
66
NA
iv315,31-DEC-14, 8
DUMMY
REF01
New Ref Project of type CPRM
66
NA
That is values are then separated by newline(when seen in wordpad)
Thanks

The defaut linesize is 80 characters, so by default columns will wrap onto new lines when that length will be exceeded. Your 100-byte columns will cause that behaviour. You can add SQL*Plus commands to change that:
-- any number at least as large as the longest possible output
set linesize 1024
set wrap off
select * is generally frowned up as the output can change unexpectedly if the table definition changes - i.e. a column is added - or if the table has columns in different orders in different environments. (Which arguably shouldn't happen with source control, and generally doesn't matter as long as you don't use *. If you list the columns you want explicitly, it isn't much extra work to concatenate them with manually-added separators, e.g:
select wwid
||','|| to_char(timelog_date, 'DD-MON-YY') -- or another format
||','|| total_hours
||','|| activity
||','|| subactivity
... -- etc
from FIMS_OWNER.TIMELOG_EXTRACT_OUTBOUND_T;
That will remove extra whitespace that is currently going to pad all the CSV rows to the same length, and reduce the output file size. The output is then a single column, so the colsep setting isn't relevant any more.

Related

Remove Blank Line In Between Select Queries When Spooling to CSV File 2

I have query that is executed from SQLPLUS (11.2.0.4) and output is spooled to file. I am getting output file empty lines after each line + many empty spaces after the last column in the query. I have opened the file in notepad++ and i can see tons of empty spaces appended to the last column and ended with CRLF ( windows) , the last column is varchar 4000.
I have read tons of articles related to the same in internet but none of them help me. I have tried to set trimspool ON/OFF, trimout ON/OFF (btw - both commands are skipped ), page 0, heading off, pagesize 0, page 0, space 0, newpage nono
Example of SQL :
set termout off
set pagesize 0
set termout off
set pagesize 0
set heading off
set feedback off
set newpage none
set space 0
set linesize 8000
set longchunksize 200000
/*above was tried step by step - no help*/
spool "G:/gggg/fffff.csv"
PROMT COL1|COL2|COL3
select col1||';'||col2||';'||nvl(col3,'') abc
FROM transactions;
spool off;
output (notepad++):
COL1|COL2|COL3CRLF
col1|col2|col3 CRLF
col1|col2|col3 CRLF
col1|col2|col3 CRLF
output(linux)
COL1|COL2|COL3$
col1|col2|$
COL3 $
col1|col2|$
COL3 $
col1|col2|$
COL3 $
I have to use SQLPLUS .
it is requested to have output file without double quotes and with tilda as delimiter
e.g.
aaaa~bbbb~cccc~eeeeeeeeeeeeeeeeeeeeeeeeee
dddd~rrrr~bggggggg~rrrrrrrrrrrrrrrrrrrrrrrrrr
eeee~rrrrrrr~ttttttt~yyyyyyyyyyyyyyyyyyyyyyyyyy
Columns type :
col1 integer
col2 varchar2(4000 BYTE)
col3 varchar2(4000 BYTE)
Current output looks like
COL1NAME~COL2NAME~COL3NAME
aaaa~bbbb~cccc~eeeeeeeeeeeeeeeeeeeeeeeeee
dddd~rrrr~bggggggg~rrrrrrrrrrrrrrrrrrrrrrrrrr
eeee~rrrrrrr~ttttttt~yyyyyyyyyyyyyyyyyyyyyyyyyy
I would like to remove the empty lines without any cut of the characters in lines.
I can see in notepad++ :
COL1NAME~COL2NAME~COL3NAMECRLF
aaaa~bbbb~cccc~eeeeeeeeeeeeeeeeeeeeeeeeee..................................CRLF
dddd~rrrr~bggggggg~rrrrrrrrrrrrrrrrrrrrrrrrrr..............................CRLF
eeee~rrrrrrr~ttttttt~yyyyyyyyyyyyyyyyyyyyyyyyyy............................CRLF
I can see in linux (after less) :
COL1NAME~COL2NAME~COL3NAME$
aaaa~bbbb~cccc~eeeeeeeeeeeeeeeeeeee$
eeeeee $
$
dddd~rrrr~bggggggg~rrrrrrrrrrrrrrrr$
rrrrrrrrrr $
$
eeee~rrrrrrr~ttttttt~yyyyyyyyyyyyyy$
yyyyyyyyyyyy $
$
What's your table look like? Is COL3 defined as a CHAR or VARCHAR2? If a CHAR, that explains everything.
We need to see your data and your table, to be able to help you, otherwise you're asking us to guess.
Using the HR sample schema and the locations table, notice the varchar2's.
CREATE TABLE locations (
location_id NUMBER(4, 0),
street_address VARCHAR2(40),
postal_code VARCHAR2(12),
city VARCHAR2(30)
CONSTRAINT loc_city_nn NOT NULL ENABLE,
state_province VARCHAR2(25),
country_id CHAR(2),
CONSTRAINT loc_c_id_fk FOREIGN KEY ( country_id )
REFERENCES hr.countries ( country_id )
ENABLE
);
COUNTRY_ID is CHAR(2) - so you'll always see 2 spaces for that column, even if you only inserted '1' character.
Querying the table, with sqlformat set to CSV, I get CSV back, automatically.
SQLPlus 11.2.0.4 is approx from 2008, that's quite old.
Here's what you can do with Oracle SQLcl - smaller than SQLPlus, more powerful, and more user-friendly. Also, less code to get what you want.
SQLcl: Release 21.4 Production on Wed Feb 16 08:40:23 2022
Copyright (c) 1982, 2022, Oracle. All rights reserved.
Last Successful login time: Wed Feb 16 2022 08:40:24 -05:00
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
login.sql found in the CWD. DB access is restricted for login.sql.
Adjust the SQLPATH to include the path to enable full functionality.
SQL> set pagesize 200
SQL> cd c:\users\jdsmith\desktop
SQL> spool peeps.csv
SQL> select * from locations;
"LOCATION_ID","STREET_ADDRESS","POSTAL_CODE","CITY","STATE_PROVINCE","COUNTRY_ID"
1000,"1297 Via Cola di Rie","00989","Roma","","IT"
1100,"93091 Calle della Testa","10934","Venice","","IT"
1200,"2017 Shinjuku-ku","1689","Tokyo","Tokyo Prefecture","JP"
1300,"9450 Kamiya-cho","6823","Hiroshima","","JP"
1400,"2014 Jabberwocky Rd","26192","Southlake","Texas","US"
1500,"2011 Interiors Blvd","99236","South San Francisco","California","US"
1600,"2007 Zagora St","50090","South Brunswick","New Jersey","US"
1700,"2004 Charade Rd","98199","Seattle","Washington","US"
1800,"147 Spadina Ave","M5V 2L7","Toronto","Ontario","CA"
1900,"6092 Boxwood St","YSW 9T2","Whitehorse","Yukon","CA"
2000,"40-5-12 Laogianggen","190518","Beijing","","CN"
2100,"1298 Vileparle (E)","490231","Bombay","Maharashtra","IN"
2200,"12-98 Victoria Street","2901","Sydney","New South Wales","AU"
2300,"198 Clementi North","540198","Singapore","","SG"
2400,"8204 Arthur St","","London","","UK"
2500,"Magdalen Centre, The Oxford Science Park","OX9 9ZB","Oxford","Oxford","UK"
2600,"9702 Chester Road","09629850293","Stretford","Manchester","UK"
2700,"Schwanthalerstr. 7031","80925","Munich","Bavaria","DE"
2800,"Rua Frei Caneca 1360 ","01307-002","Sao Paulo","Sao Paulo","BR"
2900,"20 Rue des Corps-Saints","1730","Geneva","Geneve","CH"
3000,"Murtenstrasse 921","3095","Bern","BE","CH"
3100,"Pieter Breughelstraat 837","3029SK","Utrecht","Utrecht","NL"
3200,"Mariano Escobedo 9991","11932","Mexico City","Distrito Federal,","MX"
23 rows selected.
SQL> spool off
SQL>
And here's what that looks like -
If you can update your question with sample DDL and data, we can give you a better answer.
Disclaimer: I'm a product manager at Oracle for our Database tools.

Getting the last character in an Oracle CLOB column

I am having an issue with getting the last character from an Oracle CLOB column. The same SQL that works on an VARCHAR column doesn't work on a CLOB column and can't figure out why or how to make it work.
(this works in getting the last character from a VARCHAR)
select substr(COLUMN_NAME, length(COLUMN_NAME), 1) from TABLE_NAME
(when applied to a CLOB column, it doesn't return the last character.
select substr(CLOB_COLUMN, length(CLOB_COLUMN), 1) from TABLE_NAME
(Updates below)
Per Alex suggestion, I added the dump/cast and get "Typ=1 Len=1: 10" for that column. (I have no idea what that is)
Here is the SQL I was running:
set linesize 32000
select
length(COLUMN_NAME) as clob_length,
substr(trim(COLUMN_NAME), length(trim(COLUMN_NAME)) -50 ) as last_50,
SUBSTR(COLUMN_NAME, -1) as last_char,
dump(cast(substr(COLUMN_NAME, -1) as varchar2(1)))
from TABLENAME
I added the SQL I used and the results for one of my rows are:
CLOB_LENGTH: 1227
LAST_50: shall be permitted to be continued in service
LAST_CHAR:
VARCHAR2(1): Typ=1 Len=1: 10
(I am not clear as to what the last value means)
Works for my simple test case:
SQL> create table test (col clob);
Table created.
SQL> insert into test (col) values ('This is a cloB');
1 row created.
SQL> select substr(col, length(col), 1) from test;
SUBSTR(COL,LENGTH(COL),1)
--------------------------------------------------------------------------------
B
Though, it is simpler to use substr(col, -1) (take the last character):
SQL> select substr(col, -1) from test;
SUBSTR(COL,-1)
--------------------------------------------------------------------------------
B
SQL>
I added the dump/cast and get "Typ=1 Len=1: 10" for that column
As suspected from your image, your CLOB values end with new lines (character 10, which is what the dump output shows), and probably other whitespace, or perhaps line feed (character 13) if the original data came from, say, Word. That's why the 'last 50' isn't showing 50 'normal' characters - there are multiple blank lines taking up some of those positions.
If you want to ignore all of those trailing characters you can change your trim to include more characters, and apply that to both substr calls; something like:
substr(rtrim(clob_column, chr(32)||chr(9)||chr(10)||chr(13)), -1)
which will remove any combination of spaces, tabs, new lines and line feeds.
db<>fiddle showing the 50 character version now appears as a single line, and the last character is null, '.', 'n', 'e' etc.

Why can't I trim a column of type CHAR?

Like the title says, if a create a table in my DB :
CREATE TABLE TEST
(
FIELD CHAR(20 CHAR) NULL
)
NOLOGGING
NOCOMPRESS
NOCACHE;
Insert this :
Insert into TEST
(FIELD)
Values
('TEST -here are blank spaces- ');
COMMIT;
Then i run the following statement :
UPDATE TEST SET FIELD = TRIM(FIELD);
COMMIT;
but the field still has blank spaces, notice that if I change the data type to varchar2, it works ... does anyone know why?
Thanks!
char is a fixed width data type. A char(20) will always and forever have a length of 20. If you try to insert a shorter string, it will be padded with spaces to the fixed width length of the field. So
UPDATE TEST SET FIELD = TRIM(FIELD);
removes the spaces due to the trim function, then adds them back because the string that gets written has to be exactly 20 bytes long.
Practically, there is almost never a case to use char. You're almost always better off with a varchar2. varchar2 is a variable length data type so there is no need for the database to append the spaces to the end.

convert oracle query output to xlsx ( excel ) format in oracle

I have the query like
select * from table ;
I want to put that result into xlsx format and schedule that query.
I have the issue in how to load oracle query result into xlsx ?
Use Spool to export the query result into csv:
set colsep , -- separate columns with a comma
set pagesize 0 -- No header rows
set trimspool on -- remove trailing blanks
set headsep off -- this may or may not be useful...depends on your headings.
set linesize X -- X should be the sum of the column widths
set numw X -- X should be the length you want for numbers (avoid scientific notation on IDs)
spool myfile.csv
select * from TABLE;
A true xlsx file is a binary file in a MS proprietary format.
sqlplus can only return character strings -- text data. 'spool' only writes what sqlplus returns -- character strings.

SQL Loader issue loading decimal negative numbers

I have the following issue using SQL Loader to load a small table, my process populates a field which can be populated for possitive or negative decimal numbers, and the problem is that when the number is possitive, the process rounds with 2 positions, but if it's negative only use 1 position. I need 2 position, and this is the simple code:
UNRECOVERABLE LOAD DATA
TRUNCATE
INTO TABLE [SCHEMA].TABLE1
FIELDS TERMINATED BY '|'
OPTIONALLY ENCLOSED BY '"'
TRAILING NULLCOLS
(
FIELD1 "trim(:FIELD1 )"
)
Example:
Source: 12345,78 Target: 1234,78
Source: -12345,78 Target: -1234,8
Edit1: It just happend with number with 7 or more integer length (X < -1000000), but I can insert this numbers with regular insert instead of SQLLoader.
Edit2: I have noticed that it is not a problem with sql*loader because the .ctl is already wrong. This file is generated with python and SQL with this properties:
set colsep |
set headsep off
set pagesize 0
set trimspool on
set linesize 10000
set termout off
set feedback off
set arraysize 250
But I dont know where it is defined the lenght of each field, and this looks the error.
I'd say that you're not looking correctly. Data is, probably, stored OK, but you don't see it. Here's an example:
SQL> create table test (field1 number);
Table created.
SQL>
Control file:
load data
infile *
replace into table test
fields terminated by '|' trailing nullcols
(
field1
)
begindata
12345,78
-12345,78
987654321
-987654321
987654321,01234
-987654321,01234
111333444,887766
-112333444,887766
Loading session & the result:
SQL> $sqlldr scott/tiger#orcl control=test26.ctl log=test26.log
SQL*Loader: Release 11.2.0.2.0 - Production on Uto Kol 21 08:52:10 2018
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
Commit point reached - logical record count 7
Commit point reached - logical record count 8
SQL> select * from test;
FIELD1
----------
12345,78
-12345,78
987654321
-987654321
987654321
-987654321
111333445
-112333445
8 rows selected.
SQL>
Ooops! Seems to be wrong, eh? But, if you fix the column format ...
SQL> col field1 format 999g999g999g990d000000
SQL> select * from test;
FIELD1
-----------------------
12.345,780000
-12.345,780000
987.654.321,000000
-987.654.321,000000
987.654.321,012340
-987.654.321,012340
111.333.444,887766
-112.333.444,887766
8 rows selected.
SQL>
See? Everything is here.
I guess that, if you used a GUI, you wouldn't have such problems. The former sentence doesn't mean that you should NOT use SQL*Plus - on the contrary; just ... learn how to use it properly :)
[EDIT, after reading the comment]
It is SET NUMFORMAT you might be looking for:
SQL> select -12334.789123 from dual;
-12334.789123
-------------
-12334,789
SQL> set numformat 999g999g990d000000
SQL> select -12334.789123 from dual;
-12334.789123
-------------------
-12.334,789123
SQL>

Resources