This is my query in hive
select colname from uber_test;
OK
{"data":"{\"age\":42, \"gender\":\"male\"}"}
This is how I export data:
CREATE TABLE export_test_1( name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE;
INSERT OVERWRITE LOCAL DIRECTORY 'export_test_1' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' select colname from uber_test;
This is what my data looks like after I export:
more export_test_2/000000_0
data{"age":42, "gender":"male"}
I need to preserve the json. needs to be:
{"data":"{\"age\":42, \"gender\":\"male\"}"}
Related
In Hive, the statement below will output foo^Bbar^Abaz
insert overwrite directory 's3://bucket-name/foobarbaz'
row format delimited
fields terminated by '\001'
select split('foo,bar', ','), 'baz';
In Presto, I ran this statement:
insert overwrite directory 's3://bucket-name/foobarbaz'
select split('foo,bar', ','), 'baz';
With this result: ["foo","bar"]^Abaz
What is the equivalent Presto clause for insert overwrite directory that works for arrays and structs?
It seems like Presto converted my array type into a json string, but I want this formatted to Hadoop spec with collection item and map key delimiter support.
Try to specify COLLECTION ITEMS TERMINATED BY in the create table DDL.
row_format DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]] [COLLECTION ITEMS TERMINATED BY char] ...
I am new to Hive and am creating external tables on csv file. One of the issues I am coming across are values that contain multiple commas within string itself. For example, the csv file contains the following:
CSV File
When I create an external table in Hive, because there are columns within the "name" column, it shifts the first name to the right adding another column. This throws all of the data off when you view the table in Hive.
External Table result in Hive
Is there anything I can add to my script to keep the commas but also keep first and last name in the same column when the external table is created? Thank you all in advance - I am very new to Hive.
CREATE EXTERNAL TABLE database.table name (
ID INT,
Name String,
City String,
State String
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION '/xyz/xyz/database/directory/'
TBLPROPERTIES ("skip.header.line.count"="1");
Check this solution - you need to add this line : ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
https://community.cloudera.com/t5/Support-Questions/comma-in-between-data-of-csv-mapped-to-external-table-in/td-p/220193
Complete DDL example:
create table hcc(field1 string,
field2 string,
field3 string,
field4 string,
field5 string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
"separatorChar" = ",",
"quoteChar" = "\"");
I created below mention external table..
create external table if not exists sensor.building1 (BuildingID int,BuildingMgr string , BuildingAge string, HVACproduct string , Country string) row format delimited fields terminated by ',';
Loaded the table by using below query..
load data inpath '/user/cloudera/sensor/SensorFiles/building.csv' into table sensor.building1;
When I am trying to retrieve the buildingID column using below query, but I am getting null value..
select a.BuildingID
from sensor.building1 as a
limit 10;
Please guide me where I am doing something wrong
You are trying to load a CSV file into hive table but hive's default field delimiter is '\001'
So while you tring to load data from csv (I am assuming its ',' separated) its get failed.
You can create table like :
create external table test1(country string, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';
I noticed that I can use an insert into statement from text table to avro table when not using the MultiDelimitSerDe. It also works with ROW FORMAT DELIMITED FIELDS TERMINATED BY "," i.e. a single character.
I create 2 tables - 1 text table and 1 avro table:
CREATE TABLE example1 ( example STRING, example2 STRING, example3
STRING ) ROW FORMAT SERDE
'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH
SERDEPROPERTIES ("field.delim"="**") STORED AS TEXTFILE ;
CREATE TABLE example2 ( example STRING, example2 STRING, example3
STRING ) STORED AS AVRO;
I then load data into example1 table (file delimited by "**")i.e.
LOAD DATA INPATH 'HDFS-path' INTO TABLE example1;
example1 now has data inside it. I want to insert the data from example1 to example2.
INSERT INTO TABLE example2 SELECT * from example1;
This however, gives a "return code 2" error. I have no idea why I am unable to insert the data using the MultiDelimitSerDe but I am able to do this with "ROW FORMAT DELIMITED FIELDS TERMINATED BY". But, I need to use a multi-delimiter.
Could anyone help me please?
Have you added the required JAR file?
'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' - Make sure you have the required JAR file for this (hive_contrib.jar).
I am trying to load data into hive tables which is delimited by double pipe(||). When I try this :
Sample I/P:
1405983600000||111.111.82.41||806065581||session-id
Creating table in hive:
create table test_hive(k1 string, k2 string, k3 string, k4 string,) row format delimited fields terminated by '||' stored as textfile;
Loading data from text file:
load data local inpath '/Desktop/input.txt' into table test_hive;
When I do this it is storing data in the below format:
1405983600000 tabspace-as-second-column 111.111.82.41 tabspace-as-fourth-column
Where as I am expecting the data in table to be
1405983600000 111.111.82.41 806065581 session-id
Kindly help me out I have tried different options on this but unable to resolve it
Multicharater delimiter eg. || is not supported in Hive till ver 0.13 . So fields terminated by || won't work out.There is an alter native for this.
CREATE EXTERNAL TABLE page_view(viewTime INT, userid BIGINT,
page_url STRING, referrer_url STRING,
ip STRING COMMENT 'IP Address of the User',
country STRING COMMENT 'country of origination')
COMMENT 'This is the staging page view table'
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\054'
SERDE serde_name WITH SERDEPROPERTIES (field.delim='||')
STORED AS TEXTFILE
LOCATION '<hdfs_location>';
The default serde can be used. Multi character delimiters can be used for fields , line , escape characters by specifying them in the serde properties.
This issue has been resolved in hive 14 with the use of multidelimiter serde. Please find documentation here.
https://cwiki.apache.org/confluence/display/Hive/MultiDelimitSerDe
You could do this if you don't want to use alternate serde or have earlier version of hive:
create external table my_table (line string) location /path/file;
Then create view on top:
create view my_view as select split(line,'\\|\\|')[0] as column_1
, split(line,'\\|\\|')[1] as column_2
, split(line,'\\|\\|')[2] as column_3
from my_table;
Query the view. Good luck.