removing EOL delimiter from inserting into external table -oracle - oracle
I have included notrim for rowdata column in external table as suggesterd by Alex (This is a continuation of this question,),
But now End of Line character is also appending at the rowdata column, I mean , End of line (CR-LF) is also joins at the end of rowdata.
I don't want to use substr() or translate() , since file size is around 1GB,
My external table creation process :
'CREATE TABLE ' || rec.ext_table_name || ' (ROW_DATA VARCHAR2(4000)) ORGANIZATION EXTERNAL ' ||
'(TYPE ORACLE_LOADER DEFAULT DIRECTORY ' || rec.dir_name || ' ACCESS ' || 'PARAMETERS (RECORDS ' ||
'DELIMITED by NEWLINE NOBADFILE NODISCARDFILE ' ||
'FIELDS REJECT ROWS WITH ALL NULL FIELDS (ROW_DATA POSITION(1:4000) char)) LOCATION (' || l_quote ||
'temp.txt' || l_quote || ')) REJECT LIMIT UNLIMITED'
Is there any other paramenter I can add , to remove the End-of-line character. Thanks.
EDIT 1:
My file :
Some first line with spaces at end
Some second line with spaces at end
My Ext table :
Some first line with spaces at end <EOL>
Some second line with spaces at end <EOL>
to be more clear , I will explain in java (when I assign column values to string , it is something like below),
without notrim :
rowdata[1]="Some first line with spaces at end";
rowdata[2]="Some second line with spaces at end";
with notrim:
rowdata[1]="Some first line with spaces at end \n";
rowdata[2]="Some second line with spaces at end \n";
what I want it to be :
rowdata[1]="Some first line with spaces at end ";
rowdata[2]="Some second line with spaces at end ";
the delimiter is also a part of rowdata, since no trim is specified.
EDIT2:
Line-Endings : CRLF
Platform :
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit
Production PL/SQL Release 12.1.0.1.0 - Production
"CORE 12.1.0.1.0 Production" TNS for Solaris: Version 12.1.0.1.0 -
Production NLSRTL Version 12.1.0.1.0 - Production
SELECT DUMP(ROW_DATA,1016) FROM EXT_TABLE WHERE ROWNUM = 1;
Typ=1 Len=616 CharacterSet=AL32UTF8:
41,30,30,30,30,30,30,30,30,30,30,31,30,30,30,30,37,36,36,36,44,30,30,30,30,31,32,35,30,38,31,36,32,35,30,38,31,36,31,33,34,37,30,39,44,42,20,41,30,36,31,30,30,30,30,30,30,30,30,30,30,30,30,32,30,30,4d,59,52,20,32,5a,20,30,31,36,30,30,30,31,32,31,32,33,34,30,30,30,30,30,30,30,30,30,30,30,30,30,30,30,30,52,49,42,46,50,58,30,30,30,31,30,30,30,30,30,30,30,30,31,30,36,32,38,30,31,30,32,30,30,47,20,20,20,20,53,20,20,30,30,30,30,30,30,30,30,30,30,30,20,20,20,20,20,20,20,4e,39,32,37,32,20,20,20,20,20,20,30,30,30,30,30,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,30,30,39,39,38,54,45,53,54,52,52,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,54,45,53,54,4f,50,44,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,54,45,53,54,54,52,41,4e,53,49,44,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,54,45,53,54,52,52,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,54,45,53,54,4f,50,44,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,54,45,53,54,54,52,41,4e,53,49,44,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,d
Len should be 615
Your file line endings are CRLF (suggesting the file is created in Windows?), but your database is running on Solaris. As the documentation says:
If DELIMITED BY NEWLINE is specified, then the actual value used is platform-specific. On UNIX platforms, NEWLINE is assumed to be "\n". On Windows operating systems, NEWLINE is assumed to be "\r\n".
As your database platform is Unix it's only using the LF (\n) as the record delimiter. You can either change the delimiter in your file, or change the terminated by clause to look for the Windows line-ending:
,,,
records delimited by "\r\n" nobadfile ...
If you might get files with either type of line ending and can't control that, you could add a preprocessor step to strip any that do exist. If you create an executable script file, either in the same directory as the file or (as Oracle recommends) in a different Oracle-accessible directory, say called remove_cr which contains:
/usr/bin/sed -e "s/\\r$//" $1
you can add a call to that in your external table definition, and keep the newline temrinator:
...
records delimited by newline nobadfile nodiscardfile
preprocessor 'remove_cr'
...
Make sure you read the the security warnings in the documentation though.
Demo with a temp.txt file with CRLF line endings:
create table t42_ext (
row_data varchar2(4000)
)
organization external
(
type oracle_loader default directory d42 access parameters
(
records delimited by newline nobadfile nodiscardfile
preprocessor 'remove_cr'
fields reject rows with all null fields
(
row_data position(1:4000) char notrim
)
)
location ('temp.txt')
)
reject limit unlimited;
select '<'|| row_data ||'>' from t42_ext;
'<'||ROW_DATA||'>'
--------------------------------------------------------------------------------
<Line1sometext >
<Line2sometext >
<Line3sometext >
Related
Unequal length between strings after writing to a file - using same delimiter (tab)
I have a short procedure in PL/SQL which uses UTL_FILE package to create and then write to a .txt file. Tab is saved in its own variable (v_delimiter), declared as varchar2(5), with a value of chr(9). The header is saved as a string, concatenated with the v_delimiter and then written to a file. After that, the rest of the data from an explicit cursor is also written to a file, line by line. In the end, when I open the txt file, there are unequal widths between some of the strings which make up a header. There are also unequal widths between some of the data from the cursor inside a final .txt file and I guess that shouldn't be since I am using one and the same delimiter (tab) to create a header and to create a string from a cursor. I am using UTL_FILE.put_line_nchar function to write a Unicode line to a file. I tried without declaring a delimiter as a variable, using literally chr(9) when concatenating and it is always the same result. I am out of ideas why is this happening in the final txt file. v_file_handle UTL_FILE.file_type ; v_output_path VARCHAR2 (100) := '/Path/to/File' ; v_file_header VARCHAR2 (32767) ; v_delimiter VARCHAR2 (5) := chr(9) ; v_file_handle := UTL_FILE.fopen_nchar (v_output_path,'string_1' || TO_CHAR (SYSDATE, 'dd_mm_yyyy') || '.txt','w', 32767); -- opening v_file_header :='claimFileIdentifier'|| v_delimiter || 'claimFileOpenedDate' || v_delimiter|| 'claimStatus'|| v_delimiter|| 'claimStatusDate'|| v_delimiter|| 'incidentDateTime'|| v_delimiter|| 'incidentPlace'|| v_delimiter|| 'calculationType' ... -- header UTL_FILE.put_line_nchar ( v_file_handle, v_file_header ) ; --writing header to a file FOR rec IN cursor_candidates --iterating over a cursor LOOP UTL_FILE.put_line_nchar ( v_file_handle, rec.claimFileIdentifier || v_delimiter || rec.claimFileOpenedDate || v_delimiter || rec.claimStatus || v_delimiter || rec.claimStatusDate || v_delimiter || rec.incidentDateTime || v_delimiter || rec.incidentPlace || v_delimiter ... ) ; -- writing cursor rows to a file END LOOP ; UTL_FILE.fclose ( v_file_handle ); Final txt file and unequal width between certain strings
That's kind of expected, in my opinion. Values are separated by the TAB character, but it doesn't mean that output will look "nice" when you look at it as a text file. For example, following values are separated by TAB, but they look ugly: a b c Littlefoot Scott Tiger If you e.g. imported that file into Excel and set TAB as column separator, every value would be in its own column and output would look pretty. If you wanted text file to look nice as well, you'll have to use a different approach, e.g. LPAD numeric values (IDs, salaries, ...), RPAD textual strings (names, addresses, ...), possibly SUBSTR (to cut long values short).
This relating to the functionality of the client application that you use to open the output tab-separated values (TSV) file. If I have the data: longtitle longtitle2 longtitle3 a b c 1234567890 123456789010234545 1234567788 and I open it in a basic text editor (such as Notepad) then the output is: And the columns are unequal across the rows. However, if I display the same data in an editor that supports TSV files (such as Notepad++) then it displays the same output with equal widths for the columns across the rows: Importing the data into a spreadsheet application (such as Excel or OpenOffice), then the TSV is separated into cells and the output is: And, again, the information is split into columns. Do not change how you are outputting the file; instead, find a better way of viewing the output file with an application that supports formatting tab-separated values.
Multiple lines in a column in oracle to a single row
My oracle table is as follows ( Address column having multiple lines): ID Address -------------------- 1456897 No 61 11th Street Tatabad Coimbatore - 641012 How to get the desired result as (with Address column as a single line) ? ID Address ------------------------- 1456897 No 61 , 11th Street, Tatabad Coimbatore - 641012
I don't know if your database has its newlines as \x0a or \x0d or \x0d\x0a. I therefore propose a the following solution that handles all three kind of new lines. It will however replace mutliple newlines with one ,. This might be what you want, or it might not.
select
id,
regexp_replace(
address,
'('||chr(10)||'|'||chr(13)||')+',
', ') as address,
....
from
....
remove new line character in the column - something like SELECT REPLACE(Address_column, '\n', ' ') -- \n might be also \r\n or even \r FROM table_name
Oracle Regexp to replace \n,\r and \t with space
I am trying to select a column from a table that contains newline (NL) characters (and possibly others \n, \r, \t). I would like to use the REGEXP to select the data and replace (only these three) characters with a space, " ".
No need for regex. This can be done easily with the ASCII codes and boring old TRANSLATE() select translate(your_column, chr(10)||chr(11)||chr(13), ' ') from your_table; This replaces newline, tab and carriage return with space. TRANSLATE() is much more efficient than its regex equivalent. However, if your heart is set on that approach, you should know that we can reference ASCII codes in regex. So this statement is the regex version of the above. select regexp_replace(your_column, '([\x0A|\x0B|`\x0D])', ' ') from your_table; The tweak is to reference the ASCII code in hexadecimal rather than base 10.
select translate(your_column, chr(10)||chr(11)||chr(13), ' ') from your_table; to clean it is essential to serve non-null value as params ... (oracle function basically will return null once 1 param is null, there are few excpetions like replace-functions) select translate(your_column, ' '||chr(10)||chr(11)||chr(13), ' ') from your_table; this examples uses ' '->' ' translation as dummy-value to prevent Null-Value in parameter 3
SQL*Loader: Dealing with delimiter characters in data
I am loading some data to Oracle via SQLLDR. The source file is "pipe delimited". FIELDS TERMINATED BY '|' But some records contain pipe character in data, and not as separator. So it breaks correct loading of records as it understands indata pipe characters as field terminator. Can you point me a direction to solve this issue? Data file is about 9 GB, so it is hard to edit manually. For example, Loaded row: ABC|1234567|STR 9 R 25|98734959,32|28.12.2011 Rejected Row: DE4|2346543|WE| 454|956584,84|28.11.2011 Error: Rejected - Error on table HSX, column DATE_N. ORA-01847: day of month must be between 1 and last day of month DATE_N column is the last one.
You could not use any separator, and do something like: field FILLER, col1 EXPRESSION "REGEXP_REPLACE(:field,'^([^|]*)\\|([^|]*)\\|(.*)\\|([^|]*)\\|([^|]*)\\|([^|]*)$', '\\1')", col2 EXPRESSION "REGEXP_REPLACE(:field,'^([^|]*)\\|([^|]*)\\|(.*)\\|([^|]*)\\|([^|]*)\\|([^|]*)$', '\\2')", col3 EXPRESSION "REGEXP_REPLACE(:field,'^([^|]*)\\|([^|]*)\\|(.*)\\|([^|]*)\\|([^|]*)\\|([^|]*)$', '\\3')", col4 EXPRESSION "REGEXP_REPLACE(:field,'^([^|]*)\\|([^|]*)\\|(.*)\\|([^|]*)\\|([^|]*)\\|([^|]*)$', '\\4')", col5 EXPRESSION "REGEXP_REPLACE(:field,'^([^|]*)\\|([^|]*)\\|(.*)\\|([^|]*)\\|([^|]*)\\|([^|]*)$', '\\5')", col6 EXPRESSION "REGEXP_REPLACE(:field,'^([^|]*)\\|([^|]*)\\|(.*)\\|([^|]*)\\|([^|]*)\\|([^|]*)$', '\\6')" This regexp takes six capture groups (inside parentheses) separated by a vertical bar (I had to escape it because otherwise it means OR in regexp). All groups except the third cannot contain a vertical bar ([^|]*), the third group may contain anything (.*), and the regexp must span from beginning to end of the line (^ and $). This way we are sure that the third group will eat all superfluous separators. This only works because you've only one field that may contain separators. If you want to proofcheck you can for example specify that the fourth group starts with a digit (include \d at the beginning of the fourth parenthesized block). I have doubled all backslashes because we are inside a double-quoted expression, but I am not really sure that I ought to.
It looks to me that it's not really possible for SQL*Loader to handle your file because of the third field which: can contain the delimiter, is not surrounded by quotes and is of a variable length. Instead, if the data you provide is an accurate example then I can provide a sample workaround. First, create a table with one column of VARCHAR2 with length the same as the maximum length of any one line in your file. Then just load the entire file into this table. From there you can extract each column with a query such as: with CTE as (select 'ABC|1234567|STR 9 R 25|98734959,32|28.12.2011' as CTETXT from dual union all select 'DE4|2346543|WE| 454|956584,84|28.11.2011' from dual) select substr(CTETXT, 1, instr(CTETXT, '|') - 1) as COL1 ,substr(CTETXT ,instr(CTETXT, '|', 1, 1) + 1 ,instr(CTETXT, '|', 1, 2) - instr(CTETXT, '|', 1, 1) - 1) as COL2 ,substr(CTETXT ,instr(CTETXT, '|', 1, 2) + 1 ,instr(CTETXT, '|', -1, 1) - instr(CTETXT, '|', 1, 2) - 1) as COL3 ,substr(CTETXT, instr(CTETXT, '|', -1, 1) + 1) as COL4 from CTE It's not perfect (though it may be adaptable to SQL*Loader) but would need a bit of work if you have more columns or if your third field is not what I think it is. But, it's a start.
OK, I recomend you to parse the file and replace the delimiter.
In command line in Unix/linux you should do:
cat current_file | awk -F'|' '{printf( "%s,%s,", $1, $2); for(k=3;k<NF-2;k++) printf("%s|", $k); printf("%s,%s,%s", $(NF-2),$(NF-1),$NF);print "";}' > new_file
This command will not change your current file.
Will create a new file, comma delimited, with five fields.
It splits the input file on "|" and take first, second, anything to antelast, antelast, and last chunk.
You can try to sqlldr the new_file with "," delimiter.
UPDATE:
The command can be put in a script like (and named parse.awk)
#!/usr/bin/awk
# parse.awk
BEGIN {FS="|"}
{
printf("%s,%s,", $1, $2);
for(k=3;k<NF-2;k++)
printf("%s|", $k);
printf("%s,%s,%s\n", $(NF-2),$(NF-1),$NF);
}
and you can run in this way:
cat current_file | awk -f parse.awk > new_file
ORACLE - Exporting Procedures / Packages to a file
I would like to programmatically export my Procedures / Functions and Packages into individual files (as a backup) and using Oracle 9.2. The closest solution i found was using DBMS_METADATA.GET_DDL , but how do i output the CLOB to a text file, without losing any parts (due to length or indentation) ? Or maybe do you have other solutions to backup packages or other functions individually (only the one i want, not all of them) ? Thanks
Trying to get CLOBS (and LONGS) from command line utilities like SQL*Plus always seems to give me formatting/truncation problems. My solution was to write a simple utility in a non- type checking language (Perl) that uses DBMS_METADATA to bring the CLOB back into a string.
Snippet:
...
$sthRef = $dbhRef->prepare("select dbms_metadata.get_ddl(?,?) from dual");
$sthRef->execute('PACKAGE', $thisName);
while (($thisDDL) = $sthRef->fetchrow()) {
print $thisDDL;
}
$sthRef->finish;
...
If you want to get the DDL, there really is no way except DBMS_METADATA like you already said. Usually, this kind of a backup is done with exp (or expdp), although this doesn't create a SQL file like you would get with most other DBMS systems.
SET pages 0 spool proclist.sql SELECT CASE line WHEN 1 THEN 'CREATE OR REPLACE ' || TYPE || ' ' || NAME || CHR(10) || text ELSE text END FROM user_source WHERE TYPE IN ( 'PROCEDURE','FUNCTION') ORDER BY name, line; spool OFF exit
Thanks goes for RAS , guest for his answer ,
I needed to get codes for some procedures only, so I tried the code , to find that this code truncate the code after procedure name in first line of the code and replace it with three dots '...'
so I changed the code to the following:
SELECT CASE line
WHEN 1 THEN 'CREATE OR REPLACE ' -- || TYPE || ' ' || NAME || --CHR(10) || ' ('
|| text
ELSE
text
END
FROM user_source
WHERE TYPE IN ( 'PROCEDURE') and name like 'SomeThing%'
ORDER BY name, line;
and this page
export procedures & triggers
have a very usefaul code:
connect fred/flintstone;
spool procedures_punch.lst
select
dbms_metadata.GET_DDL('PROCEDURE',u.object_name)
from
user_objects u
where
object_type = 'PROCEDURE';
spool off;
Final way to do it by using Toad Schema Browser , then select all the needed procedures and mouse right click then select export from the menu.