I use sqoop2 to extract data from MySQL. I have set the Schema name when I create a job, but when I run the job, sqoop2 uses the first Schema name in MySQL rather than I set.
this is the setting when I create a job
Schema name: investment_estimation
Table name: input_data
this is the error message when I run the job
Message: GENERIC_JDBC_CONNECTOR_0003: Unable to access metadata -
.
.
Message: Table 'api_open_platform.input_data' doesn't exist
api_open_platform is my a database in MySQL
how to solve it, help me, please
Related
I recently added a table test while getting started on HBase.
I decided to reinstall HBase due to some issues.
After reinstalling and running the HBase shell I tried:
hbase(main):004:0> list
TABLE
0 row(s) in 0.0070 seconds
=> []
So there are no tables. Now I tried to add the table test
hbase(main):005:0> create 'test', 'testfamily'
ERROR: Table already exists: test!
I took a look into the log files and found the following entry
2018-06-21 07:53:30,646 WARN [ProcedureExecutor-2]
procedure.CreateTableProcedure: The table test does not exist in meta
but has a znode. run hbck to fix inconsistencies.
I ran it and got the following
$ hbase hbck test
Table hbase:meta is okay.
Number of regions: 1
Deployed on: my_IP,16201,1529567081041
0 inconsistencies detected.
Status: OK
I'm wondering if there's a way to remove the znode by hand?
I have also faced the same issue where it was showing the following error
The table does not exist in meta but has a znode. run hbck to fix inconsistencies.
The answer is obvious in the error only.
Inconsistency is caused as the table exist in your zookeeper quorum(distributed/pseudo distributed mode) or single zookeeper node(for standalone mode) but is not present in hbase .
So the solution will be to remove the table from zookeeper node.
To do so -
Open zookeeper client. bin/zkCli.sh
You can see all the tables which are picked by zookeeper by ls /hbase/table
Try to find the table name mentioned in the error and run rmr /hbase/table/<table_name>.This will remove that table from the state of zookeeper.
Try to create table again from Hbase.It will get created without any problem.
Before creating the twitter table i added this
ADD JAR hdfs:///user/hive/warehouse/hive-serdes-1.0-SNAPSHOT.jar;
I got the following error when create the twitter table in hive:
Error while processing statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde:
com.cloudera.hive.serde.JSONSerDe
Move the Jar from HDFS to Local.
Then try to add JAR in hive terminal
Then try to use the query on Twitter Table
Ideally speaking you can add jars from both Local file system or hdfs, looks like problem could be something else here.
I would recommend to follow below sequence of steps:
List the file on hdfs to make sure it exists
hadoop fs -ls hdfs://namenode_hostname:8020/user/hive/warehouse/hive-serdes-1.0-SNAPSHOT.jar
Add the jar in the hive using full path like above and verify the
addition using list jars command in hive cli Use the serde in
hive>list jars;
create table statement with proper syntax as show here for example
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormats&SerDe
I have a multi node hadoop cluster and now I installed hive on the namenode.
Im trying to create some hive tables from files stored in hdfs but Im getting this strange error:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:hdfs://namenode-VirtualBox:9000/data/posts
/posts.tbl is not a directory or unable to create one)
hive>
But, then I tried to create a table from a file stored in hdfs with only 2kb and the table was created with success.
But when I try to create a table from a file stored in hdfs larger like 200mb, and maybe less, I got that error.
Do you know why this error can be happening?
I believe somwhere in the code the url: hdfs://namenode-VirtualBox:9000/data/posts
/posts.tbl
is parsed and the url should not have the file suffix (.tbl) should just be ".../posts"
I refer you to: Unable to Create Table in HIVE reading a CSV from HDFS
When I try to create a table in hive getting the following error message. Any idea why this is happening?
hive> create table a (x int);
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)
Following were the logs:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate key name 'TBLS_FK2' at sun.reflect.GeneratedConstructorAccessor25.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorA‌​ccessorImpl.java:45)
This might work for you.
Its mandatory to initialize the schema with SCHEMA TOOL
Command: $HIVE_HOME/bin/schematool -dbType mysql -initSchema
Note: In your case -dbType is mysql if you are using mysql as rdbms.
please refer the below link: https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool
We have launched two EMR in AWS and installed the hadoop and hive-0.11.0 in one EMR and hive-0.13.1 other one.
Everything seems to be working fine but while trying to loading data into TABLE it's giving the below error and it happening in both the Hive Servers.
ERROR MESSAGE:
An error occurred when executing the SQL command: load data inpath
's3://buckername/export/employee_1/' into table employee_2 Query
returned non-zero code: 10028, cause: FAILED: SemanticException [Error
10028]: Line 1:17 Path is not legal
''s3://buckername/export/employee_1/'': Move from:
s3://buckername/export/employee_1 to:
hdfs://XXX.XX.XXX.XX:X000/mnt/hive_0110/warehouse/employee_2 is not
valid. Please check that values for params "default.fs.name" and
"hive.metastore.warehouse.dir" do not conflict. [SQL State=42000, DB
Errorcode=10028]
I searched for the reason and mean of this message, I found this link but when tried to execute command suggested in the given link it's also giving the below error.
Command:
--service metatool -updateLocation hdfs://XXX.XX.XXX.XX:X000 hdfs://XXX.XX.XXX.XX:X000
Initializing HiveMetaTool.. HiveMetaTool:Parsing failed. Reason:
Unrecognized option: -hiveconf
Any help in this will be really appreciated.
LOAD does not support S3. It is best practice to leave data in S3 and just use it as a Hive external table instead of copying the data to HDFS. Some references http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hive-additional-features.html and When you create an external table in Hive with an S3 location is the data transfered?
If you have installed hive on your Hadoop cluster, the default storage of hive data is HDFS (hive.metastore.warehouse.dir=/user/hive/warehouse).
As a workaround you can copy the file from S3 file system to HDFS and then from HDFS load the file to hive.
Most probably we may need to modify the parameter "hive.exim.uri.scheme.whitelist=hdfs,pfile" to load the data from S3 file system.