I've created an external hive table stored in "/cntt_sondn/hive/tables/test_orc" in hdfs using command:create external table test_lab.test_orc (col1 string, col2 string) stored as orc location '/cntt_sondn/hive/tables/test_orc';. It seems ok and the directory was created successfully by user hive. Then, I use NiFi to put a orc file in the directory, there's no problem, no error or warning thrown. However, when I browse to directory on NameNode UI, there are 2 directory with same name "test_orc", ones created by hive, the other created by my user. See image below:
In addition, it seems NiFi puts my orc file in the directory owned by user ra_vtg, so the directory created by user hive is empty. Therefore, no data inserted into the hive table.
Please explain why can these strange things happen.
Related
I am using Vertica 7.2 and am trying to access ORC data in HDFS. The directory location in HDFS is '/user/<path_to_ORC_dir>/' with all the ORC files that underlie a Hive table that is stored in ORC format.
The hadoopConfDir parameter in Vertica has been set to /etc/hadoop/conf. The hadoop conf directory from the separate hadoop cluster has been copied to each node in the Vertica cluster under /etc/hadoop/conf/. I have made an external Vertica table to read from the hdfs location using:
CREATE EXTERNAL TABLE test (col1 INT, etc...) AS COPY FROM 'hdfs:///user/<path_to_ORC_dir>/*' on any node orc;
However, when I try to query from the table I get the following error
select * from test;
Error opening file [hdfs:///user/<path_to_ORC_dir>/000004_0] for read: Could not find HDFS configurations for [hdfs:///user/<path_to_ORC_dir>/000004_0]
My ORC files are named 0..._0 and the file specified in the error changes each time I query.
When I make a table using the specified file in error, instead of the entire directory, I can query the table without any problems.
CREATE EXTERNAL TABLE test1 (col1 INT, etc...) AS COPY FROM 'hdfs:///user/<path_to_ORC_dir>/000004_0' on any node orc;
select * from test1;
...correct results...
What is the cause of the HDFS configuration error when trying to read the entire directory as opposed to the single file? Also to note, I can query the Hive table that is build on 'hdfs:///user//' without any problems.
I am fairly new to Hadoop (HDFS and Hbase) and Hadoop Eco system (Hive, Pig, Impala etc.). I have got a good understanding of Hadoop components such as NamedNode, DataNode, Job Tracker, Task Tracker and how they work in tandem to store the data in efficient manner.
While trying to understand fundamentals of data access layer such as Hive, I need to understand where exactly a table’s data (created in Hive) gets stored? We can create external and internal table in Hive. As external tables can be in HDFS or any other file system, Hive doesnt store data for such tables in warehouse. What about internal tables? This table will be created as a directory on one of the data nodes on Hadoop Cluster. Once we load data in these tables from local or HDFS file system, are there further files getting created to store data in tables created in Hive?
Say for example:
A sample file named test_emp_feedback.csv was brought from local file system to HDFS.
A table (emp_feedback) was created in Hive with a structure similar to csv file structure. This lead to creation of a directory in Hadoop cluster say /users/big_data/hive/emp_feedback
Now once I create the table and load data in emp_feedback table from test_emp_feedback.csv
Is Hive going to create a copy of file in emp_feedback directory? Wont it cause data redundancy?
Creating a Managed table will create a directory with Same name as table name at Hive warehouse directory(Usually at /user/hive/warehouse/dbname/tablename).Also the table structure(Hive Metadata) is created in the metastore(RDBMS/HCat).
Before you load the data on the table, this directory(with the same name as table name under hive warehouse) is empty.
There could be 2 possible scenarios.
If the table is external the data is not copied to warehouse directory at all.
If the table is managed(not external), when you load your data to the table it is moved(not Copied) from current HDFS location to Hive warehouse directory9/user/hive/warehouse//). So this will not replicate the data.
Caution: It is always advisable to create external table unless the data is only used by hive. Dropping a managed table would delete the data from HDFS(Warehouse of HIVE).
HadoopGig
To answer you Question :
For External Tables:
Hive does not move the data into its warehouse directory. If the external table is dropped, then the table metadata is deleted but not the data.
For Internal tables
Hive moves data into its warehouse directory. If the table is dropped, then the table metadata and the data will be deleted.
For your reference
Difference between Internal & External tables:
For External Tables
External table stores files on the HDFS server but tables are not linked to the source file completely.
If you delete an external table the file still remains on the HDFS server.
As an example if you create an external table called “table_test” in HIVE using HIVE-QL and link the table to file “file”, then deleting “table_test” from HIVE will not delete “file” from HDFS.
External table files are accessible to anyone who has access to HDFS file structure and therefore security needs to be managed at the HDFS file/folder level.
Meta data is maintained on master node, and deleting an external table from HIVE only deletes the metadata not the data/file.
For Internal Tables
Stored in a directory based on settings in hive.metastore.warehouse.dir, by default internal tables are stored in the following directory /user/hive/warehouse you can change it by updating the location in the config file.
Deleting the table deletes the metadata and data from master-node and HDFS respectively.
Internal table file security is controlled solely via HIVE. Security needs to be managed within HIVE, probably at the schema level (depends on organization).
Hive may have internal or external tables, this is a choice that affects how data is loaded, controlled, and managed.
Use EXTERNAL tables when:
The data is also used outside of Hive. For example, the data files are read and processed by an existing program that doesn’t lock the files.
Data needs to remain in the underlying location even after a DROP TABLE. This can apply if you are pointing multiple schema (tables or views) at a single data set or if you are iterating through various possible schema.
Hive should not own data and control settings, directories, etc., you may have another program or process that will do those things.
You are not creating table based on existing table (AS SELECT).
Use INTERNAL tables when:
The data is temporary.
You want Hive to completely manage the life-cycle of the table and data.
Source:
HDInsight: Hive Internal and External Tables Intro
Internal & external tables in Hadoop- HIVE
It would not cause data redundancy. For managed (not external) tables Hive moves the data into its warehouse directory. In your example, the data will be moved from original location on HDFS to '/users/big_data/hive/emp_feedback'.
Be careful with the removal of the managed table, it will lead to removal data on HDFS also.
You can send data in two days
A) use LOAD DATA INPATH 'file_location_of_csv' INTO TABLE emp_feedback;
Note that this command will remove content at source directory and create a internal table
OR)
B) Use copyFromLocal or put command to copy local file into HDFS and then create external table and copy the data into table. Now data won't be moved from source. You can drop external table but still source data is available.
e.g.
create external table emp_feedback (
emp_id int,
emp_name string
)
location '/location_in_hdfs_for_csv file';
When you drop an external table, it only drops the meta data of HIVE table. Data still exists at HDFS file location.
Got it. This is what I was able to understand so far.
It all depends upon which type of table is being created and where from the file is picked up. Below are possible use cases
enter image description here
I have created an external table in Hive using following:
create external table hpd_txt(
WbanNum INT,
YearMonthDay INT ,
Time INT,
HourlyPrecip INT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
stored as textfile
location 'hdfs://localhost:9000/user/hive/external';
Now this table is created in location */hive/external.
Step-1: I loaded data in this table using:
load data inpath '/input/hpd.txt' into table hpd_txt;
the data is successfully loaded in the specified path ( */external/hpd_txt)
Step-2: I delete the table from */hive/external path using following:
hadoop fs -rmr /user/hive/external/hpd_txt
Questions:
why is the table deleted from original path? (*/input/hpd.txt is deleted from hdfs but table is created in */external path)
After I delete the table from HDFS as in step 2, and again I use show tables; It still gives the table hpd_txt in the external path.
so where is this coming from.
Thanks in advance.
Hive doesn't know that you deleted the files. Hive still expects to find the files in the location you specified. You can do whatever you want in HDFS and this doesn't get communicated to hive. You have to tell hive if things change.
hadoop fs -rmr /user/hive/external/hpd_txt
For instance the above command doesn't delete the table it just removes the file. The table still exists in hive metastore. If you want to delete the table then use:
drop if exists tablename;
Since you created the table as an external table this will drop the table from hive. The files will remain if you haven't removed them. If you want to delete an external table and the files the table is reading from you can do one of the following:
Drop the table and then remove the files
Change the table to managed and drop the table
Finally the location of the metastore for hive is by default located here /usr/hive/warehouse.
The EXTERNAL keyword lets you create a table and provide a LOCATION so that Hive does not use a default location for this table. This comes is handy if you already have data generated. Else, you will have data loaded (conventionally or by creating a file in the directory being pointed by the hive table)
When dropping an EXTERNAL table, data in the table is NOT deleted from the file system.
An EXTERNAL table points to any HDFS location for its storage, rather than being stored in a folder specified by the configuration property hive.metastore.warehouse.dir.
Source: Hive docs
So, in your step 2, removing the file /user/hive/external/hpd_txt removes the data source(data pointing to the table) but the table still exists and would continue to point to hdfs://localhost:9000/user/hive/external as it was created
#Anoop : Not sure if this answers your question. let me know if you have any questions further.
Do not use load path command. The Load operation is used to MOVE ( not COPY) the data into corresponding Hive table. Use put Or copyFromLocal to copy file from non HDFS format to HDFS format. Just provide HDFS file location in create table after execution of put command.
Deleting a table does not remove HDFS file from disk. That is the advantage of external table. Hive tables just stores metadata to access data files. Hive tables store actual data of data file in HIVE tables. If you drop the table, the data file is untouched in HDFS file location. But in case of internal tables, both metadata and data will be removed if you drop table.
After going through you helping comments and other posts, I have found answer to my question.
If I use LOAD INPATH command then it "moves" the source file to the location where external table is being created. Which although, wont be affected in case of dropping the table, but changing the location is not good. So use local inpath in case of loading data in Internal tables .
To load data in external tables from a file located in the HDFS, use the location in the CREATE table query which will point to the source file, for example:
create external table hpd(WbanNum string,
YearMonthDay string ,
Time string,
hourprecip string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
stored as textfile
location 'hdfs://localhost:9000/input/hpd/';
So this sample location will point to the data already present in HDFS in this path. so no need to use LOAD INPATH command here.
Its a good practice to store a source files in their private dedicated directories. So that there is no ambiguity while external tables are created as data is in a properly managed directory system.
Thanks a lot for helping me understand this concept guys! Cheers!
I am using hive v0.13
My data is stored in hdfs, I use create "CREATE external TABLE" to create a table for those data. Everything works fine, I can issue "select" statements. The question is under the warehouse directory (hive.metastore.warehouse.dir), I don't see any files/data get added, is this normal? I know with "external" table data will not get copy to warehouse directory but shouldn't there be table meta data be stored under there?
When you create a internal table hive creates a directory with table name under the directory you have specified in hive.metastore.warehouse.dir. For me it /apps/hive/warehouse.
Suppose you have created a table name test_tbl then there will be a directory /apps/hive/warehouse/test_tbland hive store metadata into mysql or your configured RDBMS for store metadata.and when you load data using LOAD DATA INPATH command into this directory.
But in external table you specify a location in your create statement hence hive doesn't create any directory in default warehouse directory because you have already provided the location. it just store metadata information in RDBMS
You can directly load data into that location using hdfs dfs -put command and hive will treat that data for the table which is associated with that particular directory. Hence it is expected behavior for external table.
when you create a external table Metadata will be genrally stored in the RDBMS i.e., in metastore database and the data which you insert or load will be stored in the directory.
either it is an external or managed table metadata will always be in RDBMS when you query on any table hive will actually get the table schema from metastore and data from HDFS evaluates the schema with data and displays.
So, there wont be any metadata created in warehouse for external tables.
I'd like to know how to find the mapping between Hive tables and the actual HDFS files (or rather, directories) that they represent. I need to access the table files directly.
Where does Hive store its files in HDFS?
Hive tables may not necessarily be stored in a warehouse (since you can create tables located anywhere on the HDFS).
You should use DESCRIBE FORMATTED <table_name> command.
hive -S -e "describe formatted <table_name> ;" | grep 'Location' | awk '{ print $NF }'
Please note that partitions may be stored in different places and to get the location of the alpha=foo/beta=bar partition you'd have to add partition(alpha='foo',beta='bar') after <table_name>.
The location they are stored on the HDFS is fairly easy to figure out once you know where to look. :)
If you go to http://NAMENODE_MACHINE_NAME:50070/ in your browser it should take you to a page with a Browse the filesystem link.
In the $HIVE_HOME/conf directory there is the hive-default.xml and/or hive-site.xml which has the hive.metastore.warehouse.dir property. That value is where you will want to navigate to after clicking the Browse the filesystem link.
In mine, it's /usr/hive/warehouse. Once I navigate to that location, I see the names of my tables. Clicking on a table name (which is just a folder) will then expose the partitions of the table. In my case, I currently only have it partitioned on date. When I click on the folder at this level, I will then see files (more partitioning will have more levels). These files are where the data is actually stored on the HDFS.
I have not attempted to access these files directly, I'm assuming it can be done. I would take GREAT care if you are thinking about editing them. :)
For me - I'd figure out a way to do what I need to without direct access to the Hive data on the disk. If you need access to raw data, you can use a Hive query and output the result to a file. These will have the exact same structure (divider between columns, ect) as the files on the HDFS. I do queries like this all the time and convert them to CSVs.
The section about how to write data from queries to disk is https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Writingdataintothefilesystemfromqueries
UPDATE
Since Hadoop 3.0.0 - Alpha 1 there is a change in the default port numbers. NAMENODE_MACHINE_NAME:50070 changes to NAMENODE_MACHINE_NAME:9870. Use the latter if you are running on Hadoop 3.x. The full list of port changes are described in HDFS-9427
In Hive terminal type:
hive> set hive.metastore.warehouse.dir;
(it will print the path)
It's also very possible that typing show create table <table_name> in the hive cli will give you the exact location of your hive table.
Summarize few points posted earlier,
in hive-site.xml, property hive.metastore.warehouse.dir specifies where the files located under hadoop HDFS
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
To view files, use this command:
hadoop fs -ls /user/hive/warehouse
or
http://localhost:50070
Utilities > Browse the file system
or
http://localhost:50070/explorer.html#/
tested under hadoop-2.7.3, hive-2.1.1
describe formatted <table_name>; inside hive shell.
Notice the "Location" value that shows the location of the table.
Another way to check where a specific table is stored would be execute this query on the hive interactive interface:
show create table table_name;
where table_name is the name of the subject table.
An example for the above query on 'customers' table would be something like this:
CREATE TABLE `customers`(
`id` string,
`name` string)
COMMENT 'Imported by sqoop on 2016/03/01 13:01:49'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://quickstart.cloudera:8020/user/hive/warehouse/
sqoop_workspace.db/customers'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='true',
'numFiles'='4',
'totalSize'='77',
'transient_lastDdlTime'='1456866115')
LOCATION in the example above is where you should focus on. That is your hdfs location for hive warehouse.
Don't forget to like if you like this solution. Cheers!
Hive database is nothing but directories within HDFS with .db extensions.
So, from a Unix or Linux host which is connected to HDFS, search by following based on type of HDFS distribution:
hdfs dfs -ls -R / 2>/dev/null|grep db
or
hadoop fs -ls -R / 2>/dev/null|grep db
You will see full path of .db database directories. All tables will be residing under respective .db database directories.
Hive tables are stored in the Hive warehouse directory.
By default, MapR configures the Hive warehouse directory to be /user/hive/warehouse under the root volume. This default is defined in the $HIVE_HOME/conf/hive-default.xml.
In Sandbox, you need to go for /apps/hive/warehouse/ and normal cluster /user/hive/warehouse
If you look at the hive-site.xml file you will see something like this
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/usr/hive/warehouse </value>
<description>location of the warehouse directory</description>
</property>
/usr/hive/warehouse is the default location for all managed tables.
External tables may be stored at a different location.
describe formatted <table_name> is the hive shell command which can be use more generally to find the location of data pertaining to a hive table.
In Hive, tables are actually stored in a few places. Specifically, if you use partitions (which you should, if your tables are very large or growing) then each partition can have its own storage.
To show the default location where table data or partitions will be created if you create them through default HIVE commands: (insert overwrite ... partition ... and such):
describe formatted dbname.tablename
To show the actual location of a particular partition within a HIVE table, instead do this:
describe formatted dbname.tablename partition (name=value)
If you look in your filesystem where a table "should" live, and you find no files there, it's very likely that the table is created (usually incrementally) by creating a new partition and pointing that partition at some other location. This is a great way of building tables from things like daily imports from third parties and such, which avoids having to copy the files around or storing them more than once in different places.