Preface
This script has worked previously when importing a csv file into an external table when using a local path for the created directory (with a local drive like F:\stats) instead of a UNC path (\\station-1\...)
Script
CREATE OR REPLACE DIRECTORY csvdir AS '\\Station-1\mainFolder\stats';
DROP TABLE EXT_index;
CREATE TABLE EXT_index(
SESSION_STATION NVARCHAR2(60),
"user" NVARCHAR2(20),
"type" NCHAR(4),
"date" DATE,
"hour" NVARCHAR2(8),
BATCH NUMBER,
STEP NUMBER,
VARIANTE NUMBER,
TIME_MS NUMBER,
NB_FOLDER NUMBER,
NB_DOC NUMBER,
NB_FIELDS NUMBER,
NB_FIELDS_SHOW NUMBER,
NB_FIELDS_CONFIRM NUMBER,
NB_FIELDS_EMPTY NUMBER,
NB_KEY NUMBER,
NB_CHAR NUMBER,
NB_USEFUL_CHAR NUMBER
)organization external (
type oracle_loader
default directory csvdir
access parameters (
records delimited by newline skip 1
fields terminated by ';' lrtrim
missing field values are null (
SESSION_STATION,
"user",
"type",
"date" date 'yyyy-mm-dd',
"hour",
BATCH,
STEP,
VARIANTE,
TIME_MS,
NB_FOLDER,
NB_DOC,
NB_FIELDS,
NB_FIELDS_SHOW,
NB_FIELDS_CONFIRM,
NB_FIELDS_EMPTY,
NB_KEY,
NB_CHAR,
NB_USEFUL_CHAR
)
)
location('INDEX.csv')
)
reject limit unlimited;
DROP TABLE TMP_index;
CREATE TABLE TMP_index AS(SELECT * FROM EXT_index);
Information
Database used : OracleXE 18c
The folder containing the csv we need to import is '\\Station-1\mainFolder\stats' and is situated on a remote station, different than my local computer running the database
Problem
As said previously, this script works fine when csvdir is a local path (F:\mainFolder\stats), the csv is imported into the external table and the temp table can be created from it afterwards just fine.
When csvdir is a UNC path the error is the following : KUP-04027: file name check failed: INDEX.csv
When csvdir is a network drive mapped to the previous UNC path (J:\ mapped to \\Station-1\mainFolder, such as csvdir=J:\stats) the error is the following : KUP-04040: file INDEX.csv in CSVDIR not found
Both these errors occur when trying to create TMP_index from EXT_index :
CREATE TABLE TMP_index AS(SELECT * FROM EXT_index)
Error report -
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
--> Either KUP-04027 or KUP-04040 here, depending on how csvdir was created <--
When checking in SQL Developer, EXT_index is always empty, the table doesn't seem to be created properly (i.e. the Data tab of that table doesn't even show column names)
Steps taken for attempted resolution
The folder '\\Station-1\mainFolder' is shared with 2 users having full control over it : Everyone and User1 (my account)
On the computer that has the database (My local computer) this '\\Station-1\mainFolder' is mapped to a network drive : J:\
When using J:\, csvdir is created as 'J:\stats'
When using \\Station-1\mainFolder, csvdir is created as '\\Station-1\mainFolder\stats'
As per this link both services TNSListener and OracleServiceXE are started logging in to my account, User1
Question
How can I import a csv file situated on a remote server into a local external table?
Related
i have multiple folders on my disk and each folder has pdf files (4 files in each folder). How can i insert files in each folder in oracle table rows. the folder name will make primary key (being unique social svc #). i have used code as is from this link but i get following error:-
ora-22285 non-existent directory or file for fileopen operation
ora-06512 at sys.dbns_lob line 805
i ve also granted all permissions on the directory to my user with command:-
grant all on directory blob_dir to testuser
pl tell me what am i doing wrong.
if you going to use BLOB data type then you can upload data from external file using SQL*Loader. In case you are going to use BFILE then you just need to copy files into Oracle Server file system and grant access to it via DIRECTORY object with READ privelege. BFILE provides read only access to external files via SQL.
I am trying to get all XML file names present in a directory in order to feed them to a procedure which pulls data out of those files. Could anyone help with how I can get the file name using the EXTERNAL TABLE.
I am having trouble with ACCESS PARAMETERS and LOCATION file. Don't know what exactly would go there.
Thanks
CREATE TABLE S7303786.XML_FILES
(
FILE_NAME VARCHAR2(255 CHAR)
)
ORGANIZATION EXTERNAL
(
TYPE ORACLE_LOADER
DEFAULT DIRECTORY AUTOACCEPT_XMLDIR
ACCESS PARAMETERS
(
RECORDS DELIMITED BY NEWLINE
PREPROCESSOR AUTOACCEPT_XMLDIR: 'list_file.sh'
FIELDS TERMINATED BY WHITESPACE
)
LOCATION ('list_file.sh')
)
REJECT LIMIT UNLIMITED;
list_files.sh just contains the directory where the files are present.
sticky.txt has nothing in it
error I am getting are :
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04004: error while reading file /home/transfer/stu/nshstrans/sticky.txt
Error you got might have something to do with directory, Oracle object which points to physical directory on database server's disk. It is created by a privileged user - SYS, who then grants read and/or write privileges on it to users who will use it.
If you missed to do anything of above mentioned things, your external table won't work.
So:
SQL> show user
USER is "SYS"
SQL>
SQL> create directory mydir as 'c:\temp';
Directory created.
SQL> grant read, write on directory mydir to scott;
Grant succeeded.
SQL>
Connect to Scott and create external table:
SQL> connect scott/tiger
Connected.
SQL> create table extusers
2 (username varchar2(20),
3 country varchar2(20)
4 )
5 organization external
6 (type oracle_loader
7 default directory mydir --> this is directory I created
8 access parameters
9 (records delimited by newline
10 fields terminated by ';'
11 missing field values are null
12 (username char(20),
13 country char(20)
14 )
15 )
16 location ('mydata.txt') --> name of the file that contains data
17 ) -- located in c:\temp, which is MYDIR
18 reject limit unlimited -- directory
19 /
Table created.
SQL>
Contents of the sample text file:
SQL> $type c:\temp\mydata.txt
Littlefoot;Croatia
Michel;France
Maaher;Netherlands
SQL>
Finally, let's select from the external table:
SQL> select * from extusers;
USERNAME COUNTRY
-------------------- --------------------
Littlefoot Croatia
Michel France
Maaher Netherlands
SQL>
Works OK, doesn't it? Now, try to do what I did.
On a second reading,
it appears that you don't want to read file contents, but directory contents. If that's so - apparently, it is - then see whether this helps.
In order to make it work, privileged user has to grant additional privilege - EXECUTE - to the directory.
SQL> show user
USER is "SYS"
SQL> grant execute on directory mydir to scott;
Grant succeeded.
Next step is to create an operating system executable (on MS Windows I use, it is a .bat script; on Unix, that would be a .sh, I think) which will list the directory. Note the first line - I have to navigate to a directory which is source for Oracle directory object. If you don't do that, it won't work. The .bat file is simple:
SQL> $type c:\temp\directory_contents.bat
cd c:\temp
dir /b *.txt
SQL>
Create external table:
SQL> create table extdir
2 (line varchar2(50))
3 organization external
4 (type oracle_loader
5 default directory mydir
6 access parameters
7 (records delimited by newline
8 preprocessor mydir:'directory_contents.bat'
9 fields terminated by "|" ldrtrim
10 )
11 location ('directory_contents.bat')
12 )
13 reject limit unlimited
14 /
Table created.
SQL> connect scott/tiger
Connected.
Let's see what it returns:
SQL> select * From extdir;
LINE
-----------------------------------------------
c:\Temp>dir /b *.txt
a.txt
dept.txt
emp.txt
emps.txt
externalfile1.txt
lab18.txt
mydata.txt
p.txt
parfile_01.txt
sofile.txt
test.txt
test2.txt
15 rows selected.
SQL>
Well ... yes, those are my .txt files located in c:\temp directory.
As you use *nix, I think that problem you got is related to list_files.sh script. You didn't post its contents (which would probably help - not necessarily help me as I forgot almost everything I knew about *.nix), but - regarding Preprocessing External Tables (written by Michael McLaughlin), you might need to
prepend /usr/bin before the ls, find, and sed programs: /usr/bin/ls ...
See if it helps.
I'm trying to execute this code in PL/SQL:
create or replace directory ext_tab_dir as 'C:/mydir';
GRANT READ,WRITE ON DIRECTORY ext_tab_dir TO PUBLIC;
DROP TABLE emp_load;
CREATE TABLE emp_load (v1 VARCHAR2(4000),
v2 VARCHAR2(4000)
)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER DEFAULT DIRECTORY ext_tab_dir
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
BADFILE ext_tab_dir:'bad.bad'
LOGFILE ext_tab_dir:'log.log'
FIELDS TERMINATED BY ','
)
LOCATION ('testfile.csv')
);
-- INSERT INTO tablename(v1,v2)
SELECT * From emp_load
and then getting next errors:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error error opening file C:/mydir/log.log
I do get that it has to do something with permissions, but I'm the one who created that directory, so how do I grant priveleges to myself if it is set like this by default? Is there any way to perform that sort of operation from PL/SQL?
Try something like this.
GRANT SELECT, INSERT, UPDATE, DELETE ON emp_load TO NikitaBuriak;
Replace 'NikitaBuriak' with the ID you used when you created the table..
You should grant all the privileges to the directories and files in the location pointing to the file you are trying to access.
eg : if the file you are trying to excess is in /home/dummy_folder/new_folder/file.txt
then you should grant all the administrative privileges to dummy_folder, new folder and file.txt as well
I'm trying to create an external table in Hive, but keep getting the following error:
create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/hive_test_1375711405.45852.txt";
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)
Aborting command set because "force" is false and command failed: "create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/hive_test_1375711405.45852.txt";"
The contents of /tmp/hive_test_1375711405.45852.txt are:
abc\tdef
I'm connecting via the beeline command line interface, which uses Thrift HiveServer2.
System:
Hadoop 2.0.0-cdh4.3.0
Hive 0.10.0-cdh4.3.0
Beeline 0.10.0-cdh4.3.0
Client OS - Red Hat Enterprise Linux Server release 6.4 (Santiago)
The issue was that I was pointing the external table at a file in HDFS instead of a directory. The cryptic Hive error message really threw me off.
The solution is to create a directory and put the data file in there. To fix this for the above example, you'd create a directory under /tmp/foobar and place hive_test_1375711405.45852.txt in it. Then create the table like so:
create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/foobar";
We faced similar problem in our company (Sentry, hive, and kerberos combination). We solved it by removing all privileges from non fully defined hdfs_url. For example, we changed GRANT ALL ON URI '/user/test' TO ROLE test; to GRANT ALL ON URI 'hdfs-ha-name:///user/test' TO ROLE test;.
You can find the privileges for a specific URI in the Hive database (mysql in our case).
I wanted to create an external table, but did not have the CREATE ANY DIRECTORY permission (and could not have it granted). Fair enough, I asked the DBAs to run the following:
CREATE OR REPLACE DIRECTORY ext_data_files AS '/data/ext_data_files';
GRANT ALL ON DIRECTORY ext_data_files TO MYAPPUSER;
They did, and the final object has the following script:
CREATE OR REPLACE DIRECTORY
EXT_DATA_FILES AS
'/data/ext_data_files';
GRANT READ, WRITE ON DIRECTORY SYS.EXT_DATA_FILES TO MYAPPUSER;
(I got that from asking a desc with Toad)
I was then hoping to use this directory to create my external table with the script as follows:
CREATE TABLE MYAPPUSER.MY_EXT_TABLE
(
ID VARCHAR2(100 BYTE),
LOGIN VARCHAR2(100 BYTE),
CODE VARCHAR2(100 BYTE),
CREATED_AT VARCHAR2(100 BYTE)
)
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY SYS.EXT_DATA_FILES
ACCESS PARAMETERS
( RECORDS DELIMITED BY NEWLINE
NOBADFILE
NOLOGFILE
FIELDS TERMINATED BY ';'
MISSING FIELD VALUES ARE NULL
( ID, LOGIN, CODE, CREATED_AT) )
LOCATION (SYS.EXT_DATA_FILES:'the_external_file.txt')
)
REJECT LIMIT 0
PARALLEL ( DEGREE DEFAULT INSTANCES DEFAULT )
NOMONITORING;
but then when I SELECT * FROM MY_EXT_TABLE, the result is the infamous
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-04040: file the_external_file.txt in EXT_DATA_FILES not found
ORA-06512: at "SYS.ORACLE_LOADER", line 19
(which has quite a few hits on google, but none seem related)
I'm confident of the syntax since this is the exact same script used in our DEV environment. Also, the permissions of all files and directories involved were checked and there is nothing lower than 775.
The only difference I have here from DEV (where it works) is that the directory EXT_DATA_FILES was not created by MYAPPUSER. I tried to create a synonym for it.. but had no effect.
Maybe worth mentioning, it is Oracle 10g we are talking about.
Am I missing something obvious? Is this allowed?
All directories are in fact owned by SYS. That's why there is no CREATE DIRECTORY privilege, only CREATE ANY DIRECTORY.
So try the command without prefixing the directory name with the SYS schema and see what happens.
The error message reads:
"file the_external_file.txt in EXT_DATA_FILES not found"
Are you sure it's there?