Vertica copy query and Null Value - vertica

I have a table in vertica Database, its column CON_BRWR is Varchar(100) Not Null.
I am applying encryption on this column. If this column gets empty string, that row should be rejected but due to encryption it encrypts the empty string and Null also.
To overcome this problem i tried this solution
COPY myschema.CON_BIZ ( __tempCON_BRWR FILLER VARCHAR(100),
CON_BRWR as Case WHEN __tempCON_BRWR = '' THEN NULL ELSE AESEncrypt(__tempCON_BRWR,'abcdefg') end
) FROM STDIN ENCLOSED BY '"' delimiter ',' ENFORCELENGTH SKIP 1 rejected data as table ErrorTable no commit;
that if __tempCON_BRWR filler gets empty string it should not encrypt value but in any case
it is going in else condition
I am stuck, any kind of help will be appreciated!

it is going in else condition
Your syntax a little problematic for Vertica parser, see example for right syntax.
NOTE:
But it will not solve your main problem -> reject row.
DDL:
CREATE TABLE public.haroon
(
id int,
raw varchar(100),
encrypted varchar(100)
);
Populate table with COPY FROM STDIN statement:
(most important part, take a look on CASE syntax and on definition of NULL in copy)
copy haroon(
id,
rawdata filler varchar(100),
raw as rawdata,
encrypted as case rawdata = '' or rawdata is NULL
when TRUE then NULL
else AESEncrypt(rawdata, 'secret')
end)
from stdin ENCLOSED BY '"' NULL as '' delimiter ',' direct abort on error;
daniel=> \e
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 1,""
>> 2,
>> 3,foo
>> \.
{column} as case {filler} = '' or {filler} is NULL when TRUE then NULL else AESEncrypt({filler}, {password})
COPY ... NULL AS ''
Test:
daniel=> select id, raw, encrypted::varchar(3), encrypted is NULL from haroon ;
id | raw | encrypted | ?column?
----+-----+-----------+----------
1 | | | t
2 | | | t
3 | foo | �� | f
(3 rows)
You can see that 2 first rows (id = [1,2]) contains NULL in encrypted column, data loaded without any issues.

Related

import null value (date type) from txt file to oracle by Oracle SQL developer

i have a table:
CREATE TABLE tmpTable
(
firstDay date,
lastDay date,
description char(50)
);
i have a text file include these data:
-------------------------
| 08/27/2017 null "khanh" |
| null 08/27/2017 "khanh" |
i used Oracle SQL Developer to load data from .txt file to my table but it doesn't recognize the null field
link
How can i import data in .txt file with null value
i found out the solution, just leave it withouth any space
Ex: (i replaced spacebar by '_', but in txt file it still is 08/27/2017 null "khanh").
08/27/2017_null_"khanh" => 08/27/2017__"khanh"

vertica copy command with null value for Integer

Is there any empty character i can put into the csv in order to put a null value
into an integer column
without using the ",X" pattern?
i.e. (X is a value and the first one is null)
Suppose you have a file /tmp/file.csv like this:
2016-01-10,100,abc
2016-02-21,,def
2017-01-01,300,ghi
and a target table defined as follows:
create table t1 ( dt date, id integer, txt char(10));
Then, the following command will insert NULL into "id" for the second column (the one having dt='2016-02-21'):
copy t1 from '/tmp/file.csv' delimiter ',' direct abort on error;
Now, if you want to use a special string to identify NULL values in your input file, let's say 'MYNULL':
2016-01-10,100,abc
2016-02-21,MYNULL,def
2017-01-01,300,ghi
Then... you have to run copy COPY this way:
copy t1 from '/tmp/file.csv' delimiter ',' null 'MYNULL' direct abort on error;

No data stored after Import CSV to oracle database

i want to import csv file to oracle Database.
my csv look like this :
Table A : sometimes containing 2 lines or 3 lines or empty. i want to import start Table B. so my control.ctl is :
echo OPTIONS (SKIP=5, errors=12000)
LOAD DATA
APPEND INTO TABLE xyz
when id <> '' and sales = ''
FIELDS TERMINATED BY X'09'
optionally enclosed by '"'
TRAILING NULLCOLS
(id "TRIM (:id)",
customer "TRIM (:customer)",
qty "TRIM (:qty)",
sales filler)
xyz table is :
create xyz (
id varchar2(3),
customer varchar2(255),
qty varchar2(5)
)
why no data is stored?
In Oracle, the empty string is considered null. And null is never equal to nor unequal to any other value including null. You need to use is null or is not null to look for null values.
The predicate
when id <> '' and sales = ''
can never possibly be true. No id value is ever unequal to null and no sales value is ever equal to null. Perhaps you want
when id is not null and sales is null
If you are just trying to avoid loading blank rows, however, I'd expect that your when clause would just be on id not on sales.

getting null values while loading the data from flat files into hive tables

I am getting the null values while loading the data from flat files into hive tables.
my tables structure is like this:
hive> create table test_hive (id int,value string);
and my flat file is like this:
input.txt
1 a
2 b
3 c
4 d
5 e
6 F
7 G
8 j
when I am running the below commands I am getting null values:
hive> LOAD DATA LOCAL INPATH '/home/hduser/input.txt' OVERWRITE INTO TABLE test_hive;
hive> select * from test_hive;
OK<br>
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
screen shot:
hive> create table test_hive (id int,value string);
OK
Time taken: 4.97 seconds
hive> show tables;
OK
test_hive
Time taken: 0.124 seconds
hive> LOAD DATA LOCAL INPATH '/home/hduser/input2.txt' OVERWRITE INTO TABLE test_hive;
Copying data from file:/home/hduser/input2.txt
Copying file: file:/home/hduser/input2.txt
Loading data to table default.test_hive
Deleted hdfs://hydhtc227141d:54310/app/hive/warehouse/test_hive
OK
Time taken: 0.572 seconds
hive> select * from test_hive;
OK
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
Time taken: 0.182 seconds
The default field terminator in Hive is ^A. You need to explicitly mention in your create table statement that you are using a different field separator.
Similar to what Lorand Bending pointed in the comment, use:
CREATE TABLE test_hive(id INT, value STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ';
You don't need to specify a location since you are creating a managed table (and not an external table).
Problem you are facing is because in your data the fields are separated by ' ' and while creating table you did not mention the field delimiter. So if you don't mention the field delimiter while creating hive table, by default hive considers ^A as delimiter.
So to resolve your problem, you can recreate the table mentioning the below syntax and it would work.
CREATE TABLE test_hive(id INT, value STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ';
The solution is quite simple. The Table wan't created in the right way.
Simple solution for your problem or any further problems is knowing how to load the data.
CREATE TABLE [IF NOT EXIST] mytableName(id int,value string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '/t'
STORED AS TEXTFILE ;
Now lemme explain the code :
First Line
Creating your table. The [IF NOT EXIST] is optional that tells if the table exist don't overwrite it. Its more of safety measure.
Second line
Specifies a delimiter at the table level for structured fields.
Third Item
You can include any single character, but the default is '\001'.
'/t' is for a tab space : in your case
'|' is for data which are beside each other and separated by |
' ' for one char space. And so on...
Forth Line :
Specifies the type of file in which data is to be stored. The file can be a TEXTFILE, SEQUENCEFILE, RCFILE, or BINARY SEQUENCEFILE. Or, how the data is stored can be specified as Java input and output classes.
when loading Locally :
LOCD DATA LOCAL INPATH '/your/data/path.csv' [OVERWRITE] INTO TABLE myTableName;
Always try checking your data by a simple select* statement.
Hope it helps.
Hive’s default record and field delimiters list:
\n
^A
^B
^C
press ^V^A could insert a ^A in Vim.
The elements are separated by space or tab? Let it's tab follow these steps. If separated space use ' ' instead of '\t' Ok.
hive> CREATE TABLE test_hive(id INT, value STRING) row format
delimited fields terminated by '\t' line formated by '\n' stored as filename;
Than you have to enter
hive> LOAD DATA LOCAL INPATH '/home/hduser/input.txt' OVERWRITE INTO TABLE test_hive;
hive> select * from test_hive;
Now you will get exact your expected output "filename".
please check the dataset date column it should follow the date format yyyy-mm-dd
If the string is in the form 'yyyy-mm-dd', then a date value corresponding to that year/month/day is returned. If the string value does not match this formate, then NULL is returned.
Hive Official documentation

SQL Loader Error: "Variable length field exceeds maximum length."

I have a SQL Loader Control file,
LOAD DATA
INFILE 'test.txt'
INTO TABLE TEST replace
fields terminated "|" optionally enclosed by '"' TRAILING NULLCOLS
( DOCUMENTID INTEGER(10),
CUSTID INTEGER(10),
USERID INTEGER(10),
FILENAME VARCHAR(255),
LABEL VARCHAR(50),
DESCRIPTION VARCHAR(2000),
POSTDATE DATE "YYYY-MM-DD HH24:MI:SS" NULLIF POSTDATE="",
USERFILENAME VARCHAR(50),
STORAGEPATH VARCHAR(255)
)
and it's giving me an error when I run SQL Loader on it,
Record 1: Rejected - Error on table TEST, column FILENAME.
Variable length field exceeds maximum length.
Here's that row.. the length of that column is way under 255..
1|5001572|2|/Storage/Test/5001572/test.pdf|test.pdf||2005-01-13 11:47:49||
And here's an oddity I noticed within the log file
Column Name | Position | Len | Term | Encl | Datatype
FILENAME | NEXT | 257 | | | VARCHAR
I define the length as 255 in both my table and control file. Yet the log spits it out as 257? I've tried knocking down the length in the control file to 253, so it appears as 255 in the log file, but the same issue.
Any help? This has bugged me for two days now.
Thanks.
Don't define your data fields as VARCHAR2 and INTEGER. Use CHAR. Most of the time, when loading data from a text file, you want to use CHAR, or perhaps DATE, although even that is converted from a text form. Most of the time you don't even need a length specifier. The default length for a CHAR field is 255. Your control file should look something like:
LOAD DATA
INFILE "test.txt"
INTO TABLE TEST replace
fields terminated "|" optionally enclosed by '"' TRAILING NULLCOLS
(
DOCUMENTID,
CUSTID,
USERID ,
FILENAME,
LABEL,
DESCRIPTION CHAR(2000),
POSTDATE DATE "YYYY-MM-DD HH24:MI:SS" NULLIF POSTDATE=BLANKS,
USERFILENAME,
STORAGEPATH
)
+1 for DCookie, but to expand on that it's important to distinguish between data types as specified in a table and data types in a SQL*loader control file as they mean rather different things, confusingly.
Start with a look at the the documentation, and note that when loading regular text files you need to be using the "portable" data types.
Varchar is a "non-portable" type, in which:
... consists of a binary length subfield followed by a character string of the specified length
So as DCookie says, CHAR is the thing to go for, and INTEGER EXTERNAL is a very commonly used SQL*Loader data type which you'd probably want to specify for DOCUMENTID etc.

Resources