So, i have a file A.txt
ENG,England,English
SCO,Scotland,English
IRE,Ireland,English
WAL,Wales,Welsh
i wish to load to an oracle EXTERNAL TABLES. So this is all the things that i did up til now.
CREATE DIRECTORY LOCAL_DIR AS 'C:/Directory/'
GRANT ALL ON DIRECTORY LOCAL_DIR TO ruser;
I then pasted A.txt at C:/Directory/
Then i executed following query:
CREATE TABLE countries_ext (
country_code VARCHAR2(5),
country_name VARCHAR2(50),
country_language VARCHAR2(50)
)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY LOCAL_DIR
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
MISSING FIELD VALUES ARE NULL
(
country_code,country_name,country_language
)
)
LOCATION ('Countries1.txt')
)
PARALLEL 5
REJECT LIMIT UNLIMITED;
It showed Table Created.
But when i try to execute the query:
SELECT * FROM countries_ext; I get following exception:
Unable to open file countries_ext_5244.log. The location or file does not exist.
Could someone tell me what i'm doing wrong here?
You have created your directory like this:
CREATE DIRECTORY LOCAL_DIR AS 'C:/Directory/'
But according to the Oracle documentation
the path does not include a trailing slash.
" can external tables be also used with files that are in remote machines?"
We can create directory objects against any OS directory which is addressable from the database server. So the remote machine must be mapped to a drive on the database server.
There are several issues with this. The first is that in my experience DBAs are reluctant to have their database dependent on other servers, especially ones outside their control. The second is performance: external tables cannot be tuned like heap tables, because they are just files on the local OS. Performance against a remote file will definitely be worse, because of network latency. Which brings us to the third issue, the network admin who will be concerned about the impact on the network of queries executing against a remote file. I'll be honest, I have never seen external tables being used against remote files.
Which isn't to say it can't work. But whether it's the best solution depends on how large the file is, how often you're going to query it, the capacity of your network and what your reasons are for not hosting the file on the database server.
Related
I need to analyze a large Oracle DMP file. So far, I have no experience with Oracle.
I know that the database contains information about certain people, for example a person with the name Smith.
I don't know how the database is structured (which table contains which information, are there triggers, ...).
As long as I don't know which tables I have to search, the best way I have found to work with the database files is to use grep.
This way, I can at least verify that the database really does contain the name "Smith".
Ultimately, I would like to have an SQL dump that can be viewed, filtered and understood in a text editor.
The DMP file was created with
expdp system / [PW] directory = [expdp_dir] dumpfile = [dumpfile.dmp] full = yes logfile = [logfile.log] reuse_dumpfiles = y
I know that the name Smith occurs often in the Database. Running grep -ai smith dumpfile.dmp returns many hits.
To analyze the database further I installed oracle-database and sqldeveloper-20.2.0.175.1842-x64. I imported the DMP file with
impdp USERID = system / [PW] FULL = y FILE = [dumpfile.dmp]
The folder C:\app\[user]\oradata\orcl now contains the files SYSAUX01.DBF and SYSTEM01.DBF, among others.
I suspect that these are the database files.
The command grep -ai smith * .DBF does not return any hits.
Either the files SYSAUX01.DBF and SYSTEM01.DBF are not the databases or something did not work on the import.
Using the SQL developer, I log in with the following data:
User: system
Password: [PW] (= PW from the expdp command)
SDI: orcl
In SQL developer, I do not find Smith. SQL developer displays many tables, most of which seem
to be empty and none of which I understand. I suspect that these tables are not the tables I am looking for. Perhaps I need to log in a different way (different user, different SDI?).
I tried to export the database to an SQL dump file, trying out various options that SQL developer provides,
but the result does not contain the string "Smith".
Something is not right:
Import is faulty
wrong SDI
Export is faulty
anything else
What might have gone wrong along the way?
You have a lot misconceptions in your question.
Oracle Datapump is a database utility designed for exporting and importing. But the content, either is DDL commands ( as create table, create index ) or data from the tables, is stored as binary, so you can't check the contents of those files. There are options to extract the DDL commands from the dumpfile and put it into a script.
The datafiles you are mentioned are part of the database itself, they have nothing to do with datapump. Do not touch those files
I don't know what you mean by "Smith" , if you mean an schema, after importing make a select over dba_users looking for the field username = 'SMITH'
If you mean looking for "Smith" as part of any of those tables, you will have to look in any single table of the database ( except the ones of schemas belonging to Oracle ) and for each field that is a string
SDI does not mean anything. I guess you meant SID or Oracle System ID, an unique identifier to identify a database in a specific environment
There is nothing wrong. The problem I believe is that you don't exactly know what you are looking for.
Check this
A user/schema with name SMITH
SQL> SELECT USERNAME FROM DBA_USERS WHERE USERNAME = 'SMITH' ;
A table which name contains the word SMITH ( unlikely )
SQL> SELECT TABLE_NAME FROM DBA_TABLES WHERE TABLE_NAME LIKE '%SMITH%' ;
I have a records system in an Oracle database. The records system is used to retrieve pdf, jpeg, video documents, etc.
The design is quite simple: there is a table called infrastructure_records that has a column with hyperlinks to files:
+------------+------------------------------------------+
| project_id | file_path |
+------------+------------------------------------------+
| Project 1 | X:\Records_System\Folder A\Project 1.pdf |
| Project 2 | X:\Records_System\Folder B\Project 2.jpg |
| Project 3 | X:\Records_System\Folder C\Project 3.mpg |
+------------+------------------------------------------+
We use a front-end that creates a clickable hyperlink out of the file paths. Using the hyperlink, users can navigate to the files.
Question:
I want to check the files on the network drive against the list of files in the database table.
If I could get a list of files from the network into a database table, then I could easily query for errors.
However, I'm not sure what the best way would be to create the list of files in an Oracle table.
There are approximately 60,000 files on the network drive, nestled in a variety of subfolders
The files don't exist on the same server as the Oracle database (I'm not an I.T. guy, so I'm pretty clueless about this sort of thing). However, I do have the drive mapped in Windows Explorer on the computer that I would be running the query from.
This QC operation would be run about once per month.
Performance isn't a huge concern. The list of files doesn't need to be "live", although that would certainly be a plus.
I can think of quick-and-dirty ways to do this using VBA scripting in MS Office or the like. But, it has occurred to me that there might be a more professional or out-of-box way to do this (perhaps using built-in Oracle functionality).
How can a I elegantly insert a list of files on a network drive into an Oracle table?
1) Create a batch file in (say) C:\TEMP called dirlist.bat
#echo off
dir /s /b X:\Records_System
2) Create an external table that will pre-processing using that file
CREATE TABLE dir_listing
( file_name VARCHAR2(255) )
ORGANIZATION EXTERNAL
(
TYPE ORACLE_LOADER
DEFAULT DIRECTORY temp
ACCESS PARAMETERS
(
RECORDS DELIMITED BY NEWLINE
PREPROCESSOR temp: 'dirlist.bat'
FIELDS TERMINATED BY WHITESPACE
)
LOCATION ('dummy_file.txt')
)
REJECT LIMIT UNLIMITED;
where 'temp' is an oracle directory
create directory temp as 'c:\temp';
grant read, write, execute on directory temp to ...
and 'dummy_file.txt' is any existing file
Now you just query table dir_listing like any other table to get a list of the files
I have problems with some Oracle external table
create table myExternalTable
(field1, field2....)
ORGANIZATION EXTERNAL
(TYPE ORACLE_LOADER
DEFAULT DIRECTORY myDirectory
ACCESS PARAMETERS
(RECORDS DELIMITED BY NEWLINE
NOLOGFILE
NOBADFILE
NODISCARDFILE
FIELDS TERMINATED BY '$')
LOCATION ('data.dsv'));
commit;
alter table myExternalTable reject limit unlimited; --solve reject limit reached problem
select * from myExternalTable;
When I select on the table I have this error :
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-04040: file data.dsv in myDirectory not found
It seems that the error description is not right because normally the table is already loaded with data.dsv when created.
Plus data.dsv exists in myDirectory.
What is going on? Can somebody help?
Note :
Instead of the select, this is what I normally do :
merge into myDatabaseTable
using
(select field1, field2,.... from myExternalTable) temp
on (temp.field1= myDatabaseTable.field1)
when matched then update
set myDatabaseTable.field1 = temp.field1,
myDatabaseTable.field2 = temp.field2,
......;
This works good on my development environment but on some other environment I have the error I said before :
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-04040: file data.dsv in myDirectory not found
First I thought that, in the environment it does not work, the directory did not point where it had to but selecting on dba_directories table, I could see the path of the directory is correct.
The problem is related with the access rights of the user on the operating system side. It is defined in Oracle Support Note Create Database Directory on Remote Share/Server (Doc ID 739772.1)
For my case, I created the directory with a sysdba and then for allowing other users to accesss that external table, I created another table which is creates by Create Table as Select statement for the external table. Otherwise, I need to map the Windows Oracle Service owner user to the exact Oracle user which is already defined in the note.
So, it is more like a well fitting workaround for my case.
To sum up the steps in a nutshell:
1- Create the external table T2
2- Create a table named T1 with CTAS to external table
3- Give SELECT right to T1
Hope this helps for your case.
I have a dozen web servers each writing data to a log file. At the beginning of each hour, the data from the previous hour is loaded to hive using a cron script running the command:
hive -e "LOAD DATA LOCAL INPATH 'myfile.log' INTO TABLE my_table PARTITION(dt='2015-08-17-05')"
In some cases, the command fails and exits with a code other than 0, in which case our script awaits and tries again. The problem is, in some cases of failure, the data loading does not fail, even though it shows a failure message. How can I know for sure whether or not the data has been loaded?
Example for such a "failure" where the data is loaded:
Loading data to table default.my_table partition (dt=2015-08-17-05)
Failed with exception
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter
partition. FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
Edit:
Alternatively, is there a way to query hive for the filenames loaded into it? I can use DESCRIBE to see the number of files. Can I know their names?
About "which files have been loaded in a partition":
if you had used an EXTERNAL TABLE and just uploaded your raw data
file in the HDFS directory mapped to LOCATION, then you could
(a) just run a hdfs dfs -ls on that directory from command line (or use the equivalent Java API call)
(b) run a Hive query such as select distinct INPUT__FILE__NAME from (...)
but in your case, you copy the data into a "managed" table, so there
is no way to retrieve the data lineage (i.e. which log file was used
to create each managed datafile)
...unless you add explicitly the original file name inside the log file, of
course (either on "special" header record, or at the beginning of each record - which can be done with good old sed)
About "how to automagically avoid duplication on INSERT": there is a way, but it would require quite a bit of re-engineering, and would cost you in terms of processing time /(extra Map step plus MapJoin)/...
map your log file to an EXTERNAL TABLE so that you can run an
INSERT-SELECT query
upload the original file name into your managed table using INPUT__FILE__NAME pseudo-column as source
add a WHERE NOT EXISTS clause w/ correlated sub-query, so that if the source file name is already present in target then you load nothing more
INSERT INTO TABLE Target
SELECT ColA, ColB, ColC, INPUT__FILE__NAME AS SrcFileName
FROM Source src
WHERE NOT EXISTS
(SELECT DISTINCT 1
FROM Target trg
WHERE trg.SrcFileName =src.INPUT__FILE__NAME
)
Note the silly DISTINCT that is actually required to avoid blowing away the RAM in your Mappers; it would be useless with a mature DBMS like Oracle, but the Hive optimizer is still rather crude...
I don't believe you can simply do this is in Hadoop/Hive. So here are the basics of an implementation in python:
import subprocess
x=subprocess.check_output([hive -e "select count(*) from my_table where dt='2015-08-17-05'"])
print type(x)
print x
But you have to spend some time working with backslashes to get hive -e to work using python. It can be very difficult. It may be easier to write a file with that simple query in it first, and then use hive -f filename. Then, print the output of subprocess.check_output in order to see how the output is stored. You may need to do some regex or type conversions, but I think it should just come back as a string. Then simply use an if statement:
if x > 0:
pass
else:
hive -e "LOAD DATA LOCAL INPATH 'myfile.log' INTO TABLE my_table PARTITION(dt='2015-08-17-05')"
I have setup a simple Oracle external table test that I (alongside a DBA and Unix admin) can't get to work.
The following is based on Oracle's External Tables Concepts. The database we're using is 11g.
This is the external table definition:
drop table emp_load;
CREATE TABLE emp_load
(employee_number CHAR(5),
employee_dob DATE,
employee_last_name CHAR(20),
employee_first_name CHAR(15),
employee_middle_name CHAR(15),
employee_hire_date DATE)
ORGANIZATION EXTERNAL
(TYPE ORACLE_LOADER
DEFAULT DIRECTORY defaultdir
ACCESS PARAMETERS
(RECORDS DELIMITED BY NEWLINE
FIELDS (employee_number CHAR(2),
employee_dob CHAR(20),
employee_last_name CHAR(18),
employee_first_name CHAR(11),
employee_middle_name CHAR(11),
employee_hire_date CHAR(10) date_format DATE mask "mm/dd/yyyy"
)
)
LOCATION ('external_table_test.dat')
);
This is the contents of "external_table_test.dat":
56november, 15, 1980 baker mary alice 09/01/2004
87december, 20, 1970 roper lisa marie 01/01/1999
I am able to run the script that creates "emp_load" with no issues. I can also describe the table fine. When I attempt "select * from emp_load", I get the following errors:
SQL> select * from emp_load;
select * from emp_load
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
error opening file /defaultdir/EMP_LOAD_29305.log
EDIT 1
oracle has read/write permissions on the directory.
EDIT 2
I was able to get passed this error by using the following external table definition:
CREATE TABLE emp_load
(employee_number CHAR(3),
employee_last_name CHAR(20),
employee_middle_name CHAR(15),
employee_first_name CHAR(15)
)
ORGANIZATION EXTERNAL
(TYPE ORACLE_LOADER
DEFAULT DIRECTORY defaultdir
ACCESS PARAMETERS
(RECORDS DELIMITED BY NEWLINE
BADFILE DHHSMAPSIS:'EMP.BAD'
LOGFILE DHHSMAPSIS:'EMP.LOG'
FIELDS TERMINATED BY ','
)
LOCATION ('external_table_test2.dat')
)
REJECT LIMIT UNLIMITED;
My .dat file looks like this...
056,baker,beth,mary
057,smith,teddy,john
I had to set the permissions on "EMP.BAD", "EMP.LOG" & "external_table_test2.dat" to 777 in order to get it to work. The oracle user doesn't own those files but is in the same group as the files are.
Any idea why I can't get this to work when I set the permissions on those files to 770? Again, oracle is in the same group as those files, so I figured that 770 would be OK for permissions...
Our version of Oracle is running on Red Hat Enterprise Linux. We experimented with several different types of group permissions to no avail. The /defaultdir directory had a group that was a secondary group for the oracle user. When we updated the /defaultdir directory to have a group of "oinstall" (oracle's primary group), I was able to select from the external tables underneath that directory with no problem.
So, for others that come along and might have this issue, make the directory have oracle's primary group as the group and it might resolve it for you as it did us. We were able to set the permissions to 770 on the directory and files and selecting on the external tables works fine now.
Keep in mind that it's the user that is running the oracle database that must have write permissions to the /defaultdir directory, not the user logged into oracle. Typically you're running the database as the user "Oracle". It's not the same user (necessarily) that you created the external table with.
Check your directory permissions, too.
We faced the same problem:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error error opening file /fs01/app/rms01/external/logs/SH_EXT_TAB_VGAG_DELIV_SCHED.log
In our case we had a RAC with 2 nodes. After giving write permission on the log directory, on both sides, everything worked fine.
We had this error on Oracle RAC 11g on Windows, and the solution was to create the same OS directory tree and external file on both nodes.
When you want to create an external_table, all field's name must be written in UPPERCASE.
Done.