We are using flat files.but how to skip the last records in flat files.
use the Following in your ctl File if your tail record always starts with T! Haven't tested it in a while but worked for me the last time i worked with loading a staging table.
LOAD DATA
INFILE 'file.dat' BADFILE 'file.bad' DISCARDFILE 'file.dis'
APPEND
INTO TABLE tablename
-- This will skip the tail record
WHEN (01) <> 'T'
(
column names);
You can skip the header rows using the SKIP clause but to skip the last records you will have to use the WHEN clause. Typically, your trailing records (last records) will not be identical to the other records in the file and there should be an indicator to specify that this is a trailer record. You need to construct such a condition in your control file that this condition does not get satisfied.
Here is the Oracle documentation on the WHEN clause.
http://docs.oracle.com/cd/B14117_01/server.101/b10825/ldr_control_file.htm#i1005657
Here are some examples on conditional loading.
http://www.orafaq.com/wiki/SQL*Loader#Conditional_Load
Your original post needs more detail but for completeness the control file options clause has a LOAD = n option that tells sqlldr how many rows to load. If you have 100 rows and don't want to load the last 5, specify LOAD = 95.
Related
I have a question concerning the For all Entries statement in ABAP.
I know that it functions like a Select Distinct and deletes duplicate entries.
In my case, I want to write a validation for table inputs. As you can enter more than one line, I have to check every single one of them.
Loop with Select is not an Option. The alternative is For all Entries. Sadly if f.e. the same company code needs to be validated, FoE just gives back one entry. So I have no chance to identify where the error was (if there is one).
Here is the Code for the validation:
LOOP AT extract.
IF <xact> NE empty.
READ TABLE total WITH KEY <vim_xextract_key>.
IF sy-subcs EQ 0.
MOVE <vim_total_struc> TO ls_y.
APPEND ls_y TO lt_y.
ENDIF.
ENDIF.
ENDLOOP.
SELECT bukrs
FROM t001
FOR ALL ENTRIES IN #lt_y
WHERE bukrs = #lt_y-bukrs
INTO TABLE #DATA(lt_check_bukrs).
IF lt_check_bukrs IS INITIAL.
MESSAGE 'Error in company code' TYPE 'S' DISPLAY LIKE 'E'.
vim_abort_saving = abap_true.
ENDIF.
Maybe one of you has an approach or an idea.
Thank you for all answers!
Considering that the company code table should not be too long (in any sanely configured system), I would load it into the application server and do the comparison on the ABAP layer instead of the database layer.
SELECT bukrs
FROM t001
INTO TABLE #DATA(lt_all_bukrs).
LOOP AT lt_check_bukrs REFERENCE INTO DATA #(lr_check_bukrs).
IF NOT line_exists( lt_all_bukrs[ lr_check_burks->bukrs ] )
MESSAGE |Document { lv_check_bukrs->belnr } has invalid company code { lr_check_burks->bukrs }| TYPE 'S' DISPLAY LIKE 'E'.
ENDIF.
ENLOOP.
In a different case where this is not viable because the table you want to compare with is so large it would cause a TSV_TNEW_PAGE_ALLOC_FAILED, then I would go back to where you acquired the original data and perform an OUTER JOIN with the comparison table:
SELECT bkpf~opbel,
bkpf~bukrs
FROM bseg
LEFT OUTER JOIN t001 ON bkpf-burks = t001~bukrs
INTO TABLE
WHERE t001~bukrs IS NULL.
The result table should be all financial document numbers with invalid company codes.
Should this also be impossible because the source data doesn't come from the database (manual entry, read from a file, received from a webservice, whatever...) then the last option would be to pass that data to an ABAP-Managed Database Procedure. Because SQLScript can do JOINs between database tables and tables in memory. But that does require that you are using a SAP HANA database.
I am newbie in hadoop and I have to add data into table in hive.
I have data from FIX4.4 protocol, something like this...
8=FIX.4.4<SHO>9=85<SHO>35=A<SHO>34=524<SHO>49=SSGMdemo<SHO>52=20150410-15:25:55.795<SHO>56=Trumid<SHO>98=0<SHO>108=30<SHO>554=TruMid456<SHO>10=154<SHO>
8=FIX.4.4<SHO>9=69<SHO>35=A<SHO>34=1<SHO>49=Trumid<SHO>52=20150410-15:25:58.148<SHO>56=SSGMdemo<SHO>98=0<SHO>108=30<SHO>10=093<SHO>
8=FIX.4.4<SHO>9=66<SHO>35=2<SHO>34=2<SHO>49=Trumid<SHO>52=20150410-15:25:58.148<SHO>56=SSGMdemo<SHO>7=1<SHO>16=0<SHO>10=174<SHO>
8=FIX.4.4<SHO>9=110<SHO>35=5<SHO>34=525<SHO>49=SSGMdemo<SHO>52=20150410-15:25:58.164<SHO>56=Trumid<SHO>58=MsgSeqNum too low, expecting 361 but received 1<SHO>10=195<SHO>
Firstly, what i want is, in 8=FIX.4.4 8 as column name, and FIX.4.4 as value of that column, in 9=66 9 should be column name and 66 would be value of that column and so on.... and there are so many rows in raw file like this.
Secondly, same thing for another row, and that data would append in next row of table in hive.
Now what should i do that i am not able to think.
Any help would be appriciable.
I would first create a tab-separated-file containing this data. I suggested to use a regex in the comments but if that is not your strong suit you can just split on the <SHO> tag and =. Since you did not specify the language you want to use I will suggest a 'solution' in Python.
The code below shows you how to write one of your input lines to a CSV file.
This can easily be extended to support multiple of these lines or to append lines to the CSV files once it is already created.
import csv
input = "8=FIX.4.4<SHO>9=85<SHO>35=A<SHO>34=524<SHO>49=SSGMdemo<SHO>52=20150410-15:25:55.795<SHO>56=Trumid<SHO>98=0<SHO>108=30<SHO>554=TruMid456<SHO>10=154<SHO>"
l = input.split('<SHO>')[:-1] # Don't include last element since it's empty
list_of_pairs = map(lambda x: tuple(x.split('=')),l)
d = dict(list_of_pairs)
with open('test.tsv', 'wb') as c:
cw = csv.writer(c, delimiter='\t')
cw.writerow(d.keys()) # Comment this if you don't want to have a header
cw.writerow(d.values())
What this code does is first split the input line on <SHO> meaning it creates a list of col=val strings. What I does next is create a list of tuple pairs where each tuple is (col,val).
Then it creates a dictionary from this, which is not strictly necessary but might help you if you want to extend the code for more lines.
Next I create a tab-separated-value file test.tsv containing a header and the values in the next line.
This means now you have a file which Hive can understand.
I am sure you can find a lot of articles on importing CSV or tab-separated-value files, but I will give you an example of a generic Hive query you can use to import this file once it is in HDFS.
CREATE TABLE if not exists [database].[table]
([Col1] Integer, [Col2] Integer, [Col3] String,...)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
TBLPROPERTIES('skip.header.line.count'='1');
LOAD DATA inpath '[HDFS path]'
overwrite INTO TABLE [database].[table];
Hope this gives you a better idea on how to proceed.
Copy the file to HDFS and create an external table with a single column (C8), then use the below select statement to extract each columns
create external table tablename(
c8 string )
STORED AS TEXTFILE
location 'HDFS path';
select regexp_extract(c8,'8=(.*?)<SHO>',1) as c8,
regexp_extract(c8,'9=(.*?)<SHO>',1) as c9,
regexp_extract(c8,'35=(.*?)<SHO>',1) as c35,
regexp_extract(c8,'34=(.*?)<SHO>',1) as c34,
regexp_extract(c8,'49=(.*?)<SHO>',1) as c49,
regexp_extract(c8,'52=(.*?)<SHO>',1) as c52,
regexp_extract(c8,'56=(.*?)<SHO>',1) as c56,
regexp_extract(c8,'98=(.*?)<SHO>',1) as c98,
regexp_extract(c8,'108=(.*?)<SHO>',1) as c108,
regexp_extract(c8,'554=(.*?)<SHO>',1) as c554,
regexp_extract(c8,'35=(.*?)<SHO>',1) as c10
from tablename
Below is my Table structure and .CTL FILE & .CSV file while loading data i am always getting error on first row & other data is getting is getting loaded. if i left a complete blank line on first record all data gets inserted.
can you please help us why i am getting error on first record.
TABLE_STRUCTURE
ING_DATA
(
ING_COMPONENT_ID NUMBER NOT NULL,
PARENT_ING_ID NUMBER NOT NULL,
CHILD_ING_ID NUMBER NOT NULL,
PERCENTAGE NUMBER(7,4) NOT NULL
);
CTL FILE
LOAD DATA
INFILE 'C:\Users\pramod.uthkam\Desktop\Apex\Database\SQL LOADER-PROD\ING_COMPONENT\ingc.csv'
BADFILE 'D:\SQl Loader\bad_orders.txt'
INTO TABLE ING_data
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
TRAILING NULLCOLS
(
ING_Component_ID ,
Parent_ING_ID ,
Child_ING_ID ,
Percentage
)
CSV FILE
1,3,4,95.0000
2,3,5,5.0000
3,6,7,5.0000
4,6,4,95.0000
5,18,19,19.0000
6,18,20,80.0000
7,18,21,1.0000
8,34,35,85.0000
LOG FILE
Record 1: Rejected - Error on table ING_COMPONENT, column ING_COMPONENT_ID.
ORA-01722: invalid number
Table ING_COMPONENT:
7 Rows successfully loaded.
1 Row not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Space allocated for bind array: 66048 bytes(64 rows)
Read buffer bytes: 1048576
Total logical records skipped: 0
Total logical records read: 7
Total logical records rejected: 1
Total logical records discarded: 0
BAD FILE
1,3,4,95.0000
I tried loading your file by creating it as is. For me it runs fine. All 8 rows got loaded. No issues. I was trying in Red Hat Linux.
Then I tried 2 things.
dos2unix myctl.ctl
Ran SQLLDR. All rows got inserted.
Then I tried:
unix2dos myctl.ctl
RAN SQLLDR again, all 8 records rejected. So what I believe is that, your first record line ending is not as per you sqlplus readable format. When you enter a blank line manually, your default environment (like in my case UNIX) creates the correct line ending, and the records get read. I'm not sure, but I assume this, based on my own try as above.
So lets say you are loading this file in windows(I think this because your path looks like :)) In your csv file, give a blank line in beginning, then remove it, and do the same thing after first record also(give a blank line after first record, then remove it). Then try again. Maybe it works if the issue is what I suspect.
I have a fixed length data file a.dat with below data in it
1234544550002200011000330006600000
my focus is on specific positions
POSITION(1:4)
POSITION(5:8)
and I want to add values in these 2 positions and insert it in a field named Qty in XYZ_Table.
I am trying to the following in my CTL file. But it fails, and I don't know how to pursue it further.
LOAD DATA
INFILE '$SOME_DATA/a.dat'
APPEND
PRESERVE BLANKS
INTO TABLE XYZ_Table
(QTY POSITION(1:4)+POSITION(5:8) "to_number(:QTY)")
I need to achieve this addition functionality in SQL Loader only.
If the above methodology is not possible, it would be great if you can help me with a different approach.
P.S: What I am trying to achieve is just one part of the bigger CTL file.
You need to identify the positions you want to add together but not load into their own columns as "BOUNDFILLER", which means don't load them but remember them for use in an expression later. Then use like this:
LOAD DATA
infile test.dat
append
preserve blanks
INTO TABLE X_test
TRAILING NULLCOLS
(val_1 BOUNDFILLER position(1:4)
,val_2 BOUNDFILLER position(5:8)
,qty ":val_1 + :val_2"
)
If i have a CSV file that is in the following format
"fd!","sdf","dsfds","dsfd"
"fd!","asdf","dsfds","dsfd"
"fd","sdf","rdsfds","dsfd"
"fdd!","sdf","dsfds","fdsfd"
"fd!","sdf","dsfds","dsfd"
"fd","sdf","tdsfds","dsfd"
"fd!","sdf","dsfds","dsfd"
Is it possible to exclude any row where the first column has an exclamation mark at the end of the string.
i.e. it should only load the following rows
"fd","sdf","rdsfds","dsfd"
"fd","sdf","tdsfds","dsfd"
Thanks
According to the Loading Records Based on a Condition section of the SQL*Loader Control File Reference (11g):
"You can choose to load or discard a logical record by using the WHEN clause to test a condition in the record."
So you'd need something like this:
LOAD DATA ... INSERT INTO TABLE mytable WHEN mycol1 NOT LIKE '%!'
(mycol1.. ,mycol2 ..)
But the LIKE operator is not available! You only have = and !=
Maybe you could try an External Table instead.
I'd stick a CONSTRAINT on the table, and just let them be rejected. Maybe delete them after load. Or a unix "grep -v" to clear them out the file.