I am using a CTL file to load data stored in a file to a specific table in my Oracle database.
Currently, I launch the loader file using the following command line:
sqlldr user/pwd#db data=my_data_file control=my_loader.ctl
I would like to know if it is possible to use specify parameters to be retrieved in the CTL file.
Also, is it possible to retrieve the name of the data file used by the CTL to fill the table ?I also would like to insert it for each row. I currently have to call a procedure to update previously inserted records.
Any help would be appreciated !
As I know don't have any way to pass parametter as variable in ctrl. But You can use constant in ctl and modify clt file to change that constant value (in ctl file content) for every loading times.
Edit: more specific.
my_loader.ctl:
--options
load data
infile 'c:\$datfilename$' --this is optional, you can specify here or from command line
into table mytable
fields....
(
datafilename constant '$datfilename$', -- will be replace by real datafname each load
datacol1 char(1),
....
)
dataload.bat: assume that $datfilename$ is the text will be replace by datafile's name.
::sample copy
copy my_loader.ctl my_loader_temp.ctl
::replace the name of datafile (mainly the content to load into table's data column)
findandreplace my_loader_temp.ctl "$datafilename$" "%1"
::load
sqlldr user/pwd#db data=%1 control=my_loader_temp.ctl
::or with data be obmitted if you specified by infile in control file.
sqlldr user/pwd#db control=my_loader_temp.ctl
using: dataload.bat mydatafile_2010_10_10.txt
Related
I can't seem to find a way to specify a relative path for my infile when using sql loader.
I'm running it through a command line and this is what it looks like:
C:\app\...in\sqlldr.exe userid=user/pass
control="C:\User...DATA_DATA_TABLE.ctl" log="C:\User...DATA_DATA_TABLE.log"
bad = "C:\User...DATA_DATA_TABLE.bad" discard = "C:\User...DATA_DATA_TABLE.dsc"
(I've added carriage returns just for the readability on here, the command i use is one line)
And this works, it's will start inserting stuff in the table IF the path to my infile in .ctl is absolute like "C:\Usertemp\example.ldr"
My ctl was generated autmatically by sqldeveloper. And i just changed the path to this:
OPTIONS (ERRORS=50)
LOAD DATA
INFILE 'AI_SLA_DATA_DATA_TABLE.ldr' "str '{EOL}'" <-- i'm trying to get relative path here but doesn't work
APPEND
CONTINUEIF NEXT(1:1) = '#'
INTO TABLE "USER"."DATA"
...other sqldeveloper generated stuff
The .ldr file is in the same directory as the .ctl file. Is it possible to get the path of the ctl? I'm pretty sure he searches for the .ldr file next to the sqlldr.exe instead of the ctl.
Any tips to do this? I can't find answers on docs.oracle.
Thanks.
I've never tried adding a relative path to the .ctl file, but for me it works fine as a command-line argument, e.g.
C:\app\...in\sqlldr.exe userid=user/pass
control="DATA_DATA_TABLE.ctl" log="DATA_DATA_TABLE.log"
bad = "DATA_DATA_TABLE.bad" discard ="DATA_DATA_TABLE.dsc"
data="AI_SLA_DATA_DATA_TABLE.ldr"
How to export hive table/select query to csv? I have tried the command below. But it creates the output as multiple files. Any better methods?
INSERT OVERWRITE LOCAL DIRECTORY '/mapr/mapr011/user/output/'
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
SELECT fied1,field2,field3 FROM table1
Hive creates as many files as many reducers were running. This is fully parallel.
If you want single file then add order by to force running on single reducer or try to increase bytes per reducer configuration parameter:
SELECT fied1,field2,field3 FROM table1 order by fied1
OR
set hive.exec.reducers.bytes.per.reducer=67108864; --increase accordingly
Also you can try to merge files:
set hive.merge.smallfiles.avgsize=500000000;
set hive.merge.size.per.task=500000000;
set hive.merge.mapredfiles=true;
Also you can concatenate files using cat after getting them from hadoop.
You can use hadoop fs -cat /hdfspath > some.csv
command and get the output in one file.
If you want Header then you can use SED along with hive.. See this link which discusses various options in exporting Hive to CSV
https://medium.com/#gchandra/best-way-to-export-hive-table-to-csv-file-326063f0f229
i have to read a file .csv that is in a directory of my virtual box oracle. In my local system windows with sql developer i want to create a procedure that read that file in the virtual box. I don't undestand how use UTL_TCP.connection and how write that procedure
Consider using External tables.
Note that prior to creating external table You must have database directory created. The location specified is then used to read files from. You can specify separators, delimiters and filenames in the DDL statement withing creation of table. Once set-up it could be queries by using standard select statement.
For example:
create directory << db directory >> as '/u01';
create table ext_filename_csv (
column01 varchar2(4000),
column02 varchar2(4000),
column03 varchar2(4000)
)
organization external (
type oracle_loader
default directory << db directory >>
access parameters (
records delimited by newline
fields terminated by ",")
location ( << db directory >>:'filename.txt')
);
Read more here:
http://psoug.org/reference/externaltab.html
I am trying to run a hive script in pseudo distributed mode. The commands in the script runs absolutely fine when I run it interactive mode. However, when I add all those commands in a script and run I get an error.
The script:
add jar /path/to/jar/file;
create table flights(year int, month int,code string) row format serde 'com.bizo.hive.serde.csv.CSVSerde';
load data inpath '/tmp/hive-user/On_Time_On_Time_Performance_2013_1.csv' overwrite into table flights;
The 'On_Time_On_Time_Performance_2013_1.csv' does exist in the HDFS. The error I get is:
FAILED: SemanticException Line 3:17 Invalid path ''/tmp/hive-user/On_Time_On_Time_Performance_2013_1.csv'': No files matching path hdfs://localhost:54310/tmp/hive-user/On_Time_On_Time_Performance_2013_1.csv
fs.default.name=hdfs://localhost:54310
My hadoop is running fine.
Can someone give any pointers?
Thanks.
This is not really an answer, but it is a more detailed and repeatable formulation of your question.
a) one needs to download the csv-serde from here: git clone https://github.com/ogrodnek/csv-serde
b) Build it using mvn package
c) Create a text file containing three comma separated fields corresponding to the three fields of the given table.
c) If the path is say "/shared" then the following is the correct sequence to load:
add jar /shared/csv-serde/target/csv-serde-1.1.2-0.11.0-all.jar;
drop table if exists flights;
create table flights(year int, month int,code string) row format serde 'com.bizo.hive.serde.csv.CSVSerde' stored as textfile;
load data inpath '/tmp/hive-user/On_Time_On_Time_Performance_2013_1.csv' overwrite into table flights;
I do see the same error as in the OP: FAILED: SemanticException Line 2:17 Invalid path ''/tmp/hive-user/On_Time_On_Time_Performance_2013_1.csv'': No files matching path hdfs://localhost:9000/tmp/hive-user/On_Time_On_Time_Performance_2013_1.csv
Oracle DB/Windows XP:-
I am running an batch file that calls an “.ctl” file which in turn calls an “.xls” file, both present in the same folder.
The idea is to load the data onto Oracle db present on an remote oracle server.(non local machine)
I am getting this error, no matter what I do.
Oracle error:- LRM-00116: syntax error at 'control' following '='
The .bat file code is as below
rem SET SQLLOGIN=remod/P3w1d0ry#wsd
pause Ready to Load the remo.Temp_data Table
sqlldr userid=%SQLLOGIN% control=TempData.ctl errors=100
pause
The .ctl file is as follows:-
LOAD DATA
INFILE "data.xls"
replace
into table remo.Temp_data
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
TRAILING NULLCOLS
(
test_abbr "rtrim(:test_abbr)",
test_pk "rtrim(:test_pk)",
test_sk "rtrim(:test_sk)",
test_dt "rtrim(:test_dt)",
test_email "rtrim(:test_email)",
)
You've remarked out the the SET of SQLLOGIN. Also you might want to put a call in front of the sqlldr statement. You'll also need some data to load...
SET SQLLOGIN=remod/P3w1d0ry#wsd
pause Ready to Load the remo.Temp_data Table
call sqlldr userid=%SQLLOGIN% control=TempData.ctl data=mydata.csv errors=100