Loading XML data gets error saying my control file "references a non existent field" - oracle

I am getting an error loading an XML document from a text file into an Oracle table using SQL*Loader.
I have created an XML table:
CREATE TABLE TEST_XML OF XMLTYPE
XMLTYPE STORE AS SECUREFILE BINARY XML;
And I have a file test_file.xml:
<ROWSET>
<ROW>
<ID>1</ID>
<TEXT>This is some text</TEXT>
</ROW>
<ROW>
<ID>2</ID>
<TEXT>This is some more text</TEXT>
</ROW>
<ROW>
<ID>3</ID>
<TEXT>This is some other text</TEXT>
</ROW>
<ROW>
<ID>4</ID>
<TEXT>This is also some text</TEXT>
</ROW>
</ROWSET>
I have created a control file test_loading.ctl:
LOAD DATA
INFILE "test_file.xml"
append INTO TABLE TEST_XML
xmltype(XMLDATA)
(
XMLDATA LOBFILE("test_file.xml") TERMINATED BY EOF
)
When I run SQL*Loader using that control file:
sqlldr username/password control=/path/test_loading.ctl
it returns with the following error:
SQL*Loader-416: SDF clause for field XMLDATA in table TEST_XML_ARUN references a non existent field.
What am I doing wrong?

You seem to be mixing up a few ways of doing this. The error is because it's trying to interpret the "test_file.xml" inside LOBFILE() as a field reference.
If you know you will only load one XML document from a single text file you can make your control file:
LOAD DATA
INFILE *
append INTO TABLE TEST_XML
XMLType(XMLDATA)
FIELDS
(
FILL FILLER CHAR(1),
XMLDATA LOBFILE(CONSTANT test_file.xml) TERMINATED BY EOF
)
BEGINDATA
0
The BEGINDATA section has a row with a filler character for each XML doc in the file, and as there's only one, there's a single filler.
Note the CONSTANT which makes it look for a file called that, not a field. The log file shows that static name:
Table TEST_XML, loaded from every logical record.
Insert option in effect for this table: APPEND
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
FILL FIRST 1 CHARACTER
(FILLER FIELD)
XMLDATA DERIVED * EOF CHARACTER
Static LOBFILE. Filename is test_file.xml
Table TEST_XML:
1 Row successfully loaded.
0 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
To use a field you would have a data file with the file name, lets call it test_loading.dat to match the control file name, which contains:
test_file.xml
And a control file which uses that as the INFILE, and the content of its first field as the file name:
LOAD DATA
INFILE test_loading.dat
append INTO TABLE TEST_XML
XMLType(XMLDATA)
FIELDS
(
filename FILLER CHAR(30),
XMLDATA LOBFILE(filename) TERMINATED BY EOF
)
This time the log file shows the name is being retrieved dynamically:
Table TEST_XML, loaded from every logical record.
Insert option in effect for this table: APPEND
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
FILENAME FIRST 30 CHARACTER
(FILLER FIELD)
XMLDATA DERIVED * EOF CHARACTER
Dynamic LOBFILE. Filename in field FILENAME
Table TEST_XML:
1 Row successfully loaded.
0 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Read more in the documentation.
Either will work for you. As you only have a single file in your example the first version might be slightly simpler, but if you will be loading multiple files (with a table row per file) the second version is more useful.

Related

How should I write my control file which is used to load text file data into Mysql table using sqlldr command?

I have to load data from a text file into a table. My data in text file is delimited by ',' and each item is present in double quotes (i.e., "").
For example, data in the text file is like below:
"1009","John","NY","USA"
"1010","Ron","AZ","USA"
How should I write my control file in order not to include the double quotes (i.e., "") while loading data into the table.
Assuming that the table structure is like the following:
create table someTable(
colA number,
colB varchar2(100),
colC varchar2(100),
colD varchar2(100)
)
You can use the SQLLoader with a control file like:
OPTIONS(skip=0)
load data
infile "data.txt"
append into TABLE someTable
fields
terminated by ','
enclosed by '"'
(
colA "to_number(:colA)", /* here you can use a format for numbers, if any */
colB,
colC,
colD
)

SQL Loader Error while loading the data from xml file to table

I'm trying to load the data from an xml file to a table. I get the below errors, please help me out.
Table:
CREATE TABLE TEST_XML
(FILL CHAR(30)
XMLDATA CLOB);
Here is my control file
LOAD DATA
INFILE *
TRUNCATE INTO TABLE TEST_XML XMLType(XMLDATA)
FIELDS ( FILL FILLER CHAR(100), XMLDATA LOBFILE(CONSTANT test_file.xml) TERMINATED BY EOF )
BEGINDATA 0
I get the below error:
Table TEST_XML, loaded from every logical record. Insert option in
effect for this table: TRUNCATE
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- --------------------- FILL FIRST 100 CHARACTER (FILLER FIELD) XMLDATA
DERIVED * EOF CHARACTER
Static LOBFILE. Filename is test_file.xml
Record 1: Rejected - Error on table TEST_XML. ORA-01008: not all
variables bound
For me it is invalid syntax in control file. Oder of key word is relevant. Also like after begindata
LOAD DATA
INFILE *
INTO TABLE TEST_XML
truncate
FIELDS
( FILL FILLER CHAR(100)
,XMLDATA LOBFILE(CONSTANT test_file.xml) TERMINATED BY EOF )
BEGINDATA
0

How do I Insert data from text table (using MultiDelimitSerDe) to Avro Table?

I noticed that I can use an insert into statement from text table to avro table when not using the MultiDelimitSerDe. It also works with ROW FORMAT DELIMITED FIELDS TERMINATED BY "," i.e. a single character.
I create 2 tables - 1 text table and 1 avro table:
CREATE TABLE example1 ( example STRING, example2 STRING, example3
STRING ) ROW FORMAT SERDE
'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH
SERDEPROPERTIES ("field.delim"="**") STORED AS TEXTFILE ;
CREATE TABLE example2 ( example STRING, example2 STRING, example3
STRING ) STORED AS AVRO;
I then load data into example1 table (file delimited by "**")i.e.
LOAD DATA INPATH 'HDFS-path' INTO TABLE example1;
example1 now has data inside it. I want to insert the data from example1 to example2.
INSERT INTO TABLE example2 SELECT * from example1;
This however, gives a "return code 2" error. I have no idea why I am unable to insert the data using the MultiDelimitSerDe but I am able to do this with "ROW FORMAT DELIMITED FIELDS TERMINATED BY". But, I need to use a multi-delimiter.
Could anyone help me please?
Have you added the required JAR file?
'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' - Make sure you have the required JAR file for this (hive_contrib.jar).

Problem with Oracle Sql Loader control file

I'm trying to load some data using sql loader. Here is the top of my control/data file:
LOAD DATA
INFILE *
APPEND INTO TABLE economic_indicators
FIELDS TERMINATED BY ','
(ASOF_DATE DATE 'DD-MON-YY',
VALUE FLOAT EXTERNAL,
STATE,
SERIES_ID INTEGER EXTERNAL,
CREATE_DATE DATE 'DD-MON-YYYY')
BEGINDATA
01-Jan-79,AL,67.39940538,1,23-Jun-2009
... lots of other data lines.
The problem is that sql loader won't recognize the data types I'm specifying. This is the log file:
Table ECONOMIC_INDICATORS, loaded from every logical record.
Insert option in effect for this table: APPEND
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
ASOF_DATE FIRST * , DATE DD-MON-YY
VALUE NEXT * , CHARACTER
STATE NEXT * , CHARACTER
SERIES_ID NEXT * , CHARACTER
CREATE_DATE NEXT * , DATE DD-MON-YYYY
value used for ROWS parameter changed from 10000 to 198
Record 1: Rejected - Error on table ECONOMIC_INDICATORS, column VALUE.
ORA-01722: invalid number
... lots of similiar errors, expected if trying to insert char data into a numeric column.
I've tried no datatype spec, all other numeric specs, and always the same issue. Any ideas?
Also, any ideas on why it's changing the Rows parameter?
From your example, SQL*Loader will try to evaluate the string "AL" to a number value, which will result in the error message you gave. The sample data has something looking like it could be a decimal number at third position, not second as specified int he column list.

How to really skip the processing of a column?

In order to load data (from a CSV file) into an Oracle database, I use SQL*Loader.
In the table that receives these data, there is a varchar2(500) column, called COMMENTS.
For some reasons, I want to ignore this information from the CSV file.
Thus, I wrote this control file:
Options (BindSize=10000000,Readsize=10000000,Rows=5000,Errors=100)
Load Data
Infile 'XXX.txt'
Append into table T_XXX
Fields Terminated By ';'
TRAILING NULLCOLS
(
...
COMMENTS FILLER,
...
)
This code seems to work correctly, as the COMMENTS field in database is always set to null.
However, if in my CSV file I have a record where the corresponding COMMENTS field exceeds the 500 characters limit, I get an error from SQL*Loader:
Record 2: Rejected - Error on table T_XXX, column COMMENTS.
Field in data file exceeds maximum length
Is there a way to really exclude the processing of my COMMENTS fields?
I can't reproduce your problem. I'm using Oracle 10.2.0.3.0 with SQL*Loader 10.2.0.1.
Here is my test case:
SQL> CREATE TABLE test_sqlldr (
2 ID NUMBER,
3 comments VARCHAR2(20),
4 id2 NUMBER
5 );
Table created
Control file:
LOAD DATA
INFILE test.data
INTO TABLE test_sqlldr
APPEND
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
( id,
comments filler,
id2
)
data file:
1;aaa;2
3;abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz;4
5;bbb;6
I'm using the command sqlldr userid=xxx/yyy#zzz control=test.ctl and I'm getting all the rows without errors:
SQL> select * from test_sqlldr;
ID COMMENTS ID2
---------- -------------------- ----------
1 2
3 4
5 6
You may try another approach, I'm getting the same desired result with the following control file:
LOAD DATA
INFILE test.data
INTO TABLE test_sqlldr
APPEND
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
( id,
comments "substr(:comments,1,0)",
id2
)
Update following Romaintaz's comment: I looked into it again and managed to get the same error as you when the size of the column exceeded 255 characters. This is because the default datatype of SQL*Loader is char(255). If you have a column with more data you will have to specify the length. The following control file solved the problem for a column with 300 characters:
LOAD DATA
INFILE test.data
INTO TABLE test_sqlldr
APPEND
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
( id,
comments filler char(4000),
id2
)
Hope this Helps,
--
Vincent
Just to suggest a tiny improvement, you might try something like:
LOAD DATA
IN FILE test.data INTO TABLE test_sqlldr
APPEND
FIELDS TERMINATED BY ';'TRAILING NULLCOLS
(
id,
comments char(4000) "substr(:comments, 1, 200)",
id2)
Now you'll grab the first 200 characters (or any number you specify in it's place) of all comments - unless some of your input records have values for the comments field that exceed 4000 characters, in which they'll be rejected by loader with the 'exceeds max length' error noted earlier. But assuming that's rare or not the case, all the records will load with some of the comments truncated to 200 chars.
If you go over char(4000) you'll get a SQL Loader error - there's a limit to how far you can push the beast.

Resources