Can any one guide? I'm using sqljdbc4.jar from microsoft?
hduser#ubuntu:/usr/local/sqoop/lib$ sqoop list-tables --connect jdbc:sqlserver/Northwind --username sa --password xxxxxxx --driver com.microsoft.sqlserver.jdbc.SQLServerDriver
Result I'm getting is as follows
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
13/07/08 14:45:32 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/07/08 14:45:32 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
13/07/08 14:45:32 INFO manager.SqlManager: Using default fetchSize of 1000
13/07/08 14:45:33 ERROR manager.SqlManager: Error reading database metadata: java.sql.SQLException: No suitable driver found for jdbc:sqlserver/Northwind
Could not retrieve tables list from server
13/07/08 14:45:33 ERROR tool.ListTablesTool: manager.listTables() returned null
I also tried to add -libjars to the command
I believe that the issue is in invalid JDBC URL "jdbc:sqlserver/Northwind". The URL have to start at least with "jdbc:sqlserver://" (e.g. there is missing the "://" part). Please take a look into official documentation for more details.
Also in addition I would strongly recommend to drop the --driver option as it will instruct Sqoop to use different connector. Please refer to the provided log fragment for more details (line starting with "Parameter --driver is set to...").
Related
I ran sqoop command to list-databases and I am getting error.
This is the sqoop command I used:
sqoop list-databases --connect jdbc:netezza://xxxxxxxx/ --username xxxx --password xxx;
The error I am getting is :
ERROR manager.SqlManager: Generic SqlManager.listDatabases() not
implemented.
For your Information, I added Netezza jar like
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:<path to the jar>
Please share your knowledge on this. Thanks.
The list-databases syntax of sqoop works only with HSQLDB, MySQL and Oracle. Refer this note section from Sqoop User Guide.
I am trying to run a sqoop job .I am using sqoop version Sqoop 1.4.6-cdh5.8.0 and it is not working for this version
It is working fine with Sqoop 1.4.5-cdh5.4.0.
sqoop job --create E8 -- import --connect jdbc:mysql://localhost/test -- username root --password cloudera --table NAME --hive-import -m1
sqoop job --exec E8 -- --table dummy1
Is there any syntax issue.If anyone can help with this.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo
imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/12/23 04:48:10 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-
cdh5.8.0
Enter password:
16/12/23 04:48:19 INFO manager.MySQLManager: Preparing to use a
MySQL streaming resultset.
16/12/23 04:48:19 INFO tool.CodeGenTool: Beginning code generation
16/12/23 04:48:20 INFO manager.SqlManager:
Executing SQL statement: SELECT t.* FROM `NAME` AS t LIMIT 1
16/12/23 04:48:20 ERROR manager.SqlManager: Error executing
statement:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
'test.NAME' doesn't exist
Assuming you already did the basic checks (like manually placing the parameter in the job, and executing it) I would say the syntax looks to be correct.
When looking at the doc it is mentioned that one can override properties. Unfortunately, they only show an example that adds a property, and don't show overriding one.
A search led me to this open issue which leads me to believe that there is a bug that prevents you from overriding parameters properly.
Unfortunately I don't see a solution for this, some things that might help to work around the problem:
Parameterize on a different level
Play with the syntax (does it help if it is the first/last override element, what if you try to override AND add a user, what if you try to override the query parameter instead of the table parameter ...)
This seems to be a bug in sqoop-1.4.6-cdh5.8.0 and sqoop-1.4.6-cdh5.9.0
This, however, as you mentioned is working correctly with 1.4.5 version.
The below solution worked for me:
1) Download 'sqoop-1.4.5-cdh5.4.0.jar' from http://repo.spring.io/libs-release/org/apache/sqoop/sqoop/1.4.5-cdh5.4.0/
2) Replace 'sqoop-1.4.6-cdh5.8.0.jar' with 'sqoop-1.4.5-cdh5.4.0.jar' and modify the symbolic link 'sqoop.jar' to point to 'sqoop-1.4.5-cdh5.4.0.jar'
3) Though I don't support downgrading, but still this works as charm.
sqoop eval command :
sqoop eval --connect 'jdbc:mysql://<connection url>' --driver com.mysql.jdbc.Driver --query "select max(rdate) from test.sqoop_test"
gives me output:
Warning: /usr/hdp/2.3.2.0-2950/accumulo does not exist! Accumulo
imports will fail. Please set $ACCUMULO_HOME to the root of your
Accumulo installation. Warning: /usr/hdp/2.3.2.0-2950/zookeeper does
not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to
the root of your Zookeeper installation. 16/10/05 18:38:17 INFO
sqoop.Sqoop: Running Sqoop version: 1.4.6.2.3.2.0-2950 16/10/05
18:38:17 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead. 16/10/05 18:38:17
WARN sqoop.ConnFactory: Parameter --driver is set to an explicit
driver however appropriate connection manager is not being set (via
--connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
which connection manager should be used next time. 16/10/05 18:38:17
INFO manager.SqlManager: Using default fetchSize of 1000
-------------- | max(rdate) |
-------------- | 2014-01-25 |
but i want output without warning and table boundries like:
max(rdate) 2014-01-25
i basically want to store this output to a file.
thanks in advance
You can perform Sqoop Import operation to save output in HDFS.
Warnings are straight forward.
You can set $ACCUMULO_HOME, $ZOOKEEPER_HOME if available.
You can set --connection-manager corresponding to Mysql
For the sake of security,
It's recommended to use -P for password rather than writing in command.
These are not errors, you can live with these warnings.
You can create a .sh file , write your sqoop commands into it, then run it as
shell_file_name.sh > your_output_file.txt
We have two ways to get the query results:
The other way is you can write to HDFS by importing query results(--target-dir /path) and read from there.
You can change the file system option in sqoop command to store the results from import query, So the idea behind is you importing data to local file system rather HDFS.
eg: sqoop import -fs local -jt local --connect "connection string" --username root --password root query "Select * from table" --target-dir /home/output
https://sqoop.apache.org/docs/1.4.0-incubating/SqoopUserGuide.html#id1762587
Using command like:
sqoop export \
--connect jdbc:oracle:thin:'#somehostname.com:1521/prod1_adhoc' \
--username fbaggins \
--P \
--table MIDDLEEARTH \
--hcatalog-database MORDOR \
--hcatalog-table MORDOR \
--columns IS_DWARF,IS_ELF \
--verbose
Results in this error:
16/08/25 10:08:31 INFO hive.metastore: Trying to connect to metastore with URI thrift://somehostname.com:1521
16/08/25 10:08:31 INFO hive.metastore: Connected to metastore.
16/08/25 10:08:31 DEBUG util.ClassLoaderStack: Restoring classloader: sun.misc.Launcher$AppClassLoader#56aac163
16/08/25 10:08:31 ERROR tool.ExportTool: Encountered IOException running export job: java.io.IOException: java.lang.NullPointerException
16/08/25 10:08:31 DEBUG manager.OracleManager$ConnCache: Caching released connection for jdbc:oracle:thin:#somehostname.com:1521/prod1_adhoc/fbaggins
Not sure where the null is coming from as there are no nulls in the Hive table.
For reference, from hive:
hive> describe MORDOR;
OK
IS_DWARF bigint
IS_ELF string
From Oracle:
describe MORDOR
Name Null Type
----------------------- ---- -----------
IS_DWARF NUMBER(12)
IS_ELF VARCHAR2(3)
Is MORDOR a Hive view and not actually a table?
I have exactly the same problem when an object specified for sqoop --table is a view. DESCRIBE commands just prints columns of that view - so it does not show if it's a view or not. You could run SHOW CREATE TABLE MORDOR to confirm that's a view or table ("show create table" works for a view too as there is no separate command like "show create view").
SQOOP documentation explicitly states that sqoop import from a view is supported, but sqoop documentation on sqoop export for a view is silent - it's not stated if it's supported or not - it might be a SQOOP or HMS bug. Sqoop might not like what HMS returns for a view. So not exluding it might be an HMS bug.
Look at the logs again: don't you see anything absurd?
Trying to connect to metastore with URI thrift://$HOSTNAME:$PORT
Caching released connection for jdbc:oracle:thin:#$HOST:$PORT/$ALIAS/$SCHEMA
Yes. The bloody environment variables (or shell variables) have not been resolved by the shell. Although I'm not sure what happened to the Metastore URI (it's defined in a Hadoop conf file, no shell/env variables there...)
That's because you have enclosed them in SINGLE QUOTES.
--connect jdbc:oracle:thin:'#$HOST:$PORT/$ALIAS'
So you have a 3-step solution:
use double quotes whenever you want the shell to resolve variables inside
learn script shell
learn about pointers, Null pointers, and what happens when some buggy code fails to check whether an Object variable has been properly initialized (i.e. NullPointerException in Java)
I have a hadoop ha setup based on cdh5.I have tried to import tables from mysql by using sqoop failed with following error.
15/03/20 12:47:53 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic#33573e93 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic#33573e93 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
I have used the below command..
sqoop import --connect jdbc:mysql://<mysql hostname>:3306/haddata --username root --password password --table authors --hive-import
My mysql server version is 5.1.73-3. and used 5.1.34 and 5.1.17 version of mysql-connector-java
sqoop version is 1.4.5-cdh5.3.2
Please let me know any suggestion/comments.
Try including the option --driver com.mysql.jdbc.Driver in the import command.
Try using the below modified command, which can suit your purpose
sqoop import --connect jdbc:mysql://<mysql hostname>:3306/haddata --driver com.mysql.jdbc.Driver --username root --password password --table authors --hive-import
follow this link
Include the driver argument --driver com.mysql.jdbc.Driver in sqoop command.
sqoop import --connect jdbc:mysql://<mysql hostname>:3306/<db name> --username **** --password **** --table <table name> --hive-import --driver com.mysql.jdbc.Driver
The --driver parameter forces sqoop to use the latest mysql-connector-java.jar installed for mysql db on the sqoop machine
Try with mysql-connector-java-5.1.31.jar, it is compatable with sqoop 1.4.5.
mysql-connector-java-5.1.17.jar driver does not work with sqoop 1.4.5.
refer :
https://issues.apache.org/jira/browse/SQOOP-1400
If you have com.mysql.jdbc_5.1.5.jar or any version of com.mysql.jdbc_5.X.X.jar file in $HADOOP_HOME/bin folder, then remove that, and execute your SQOOP query.
including the option --driver com.mysql.jdbc.Driver in the import command worked for me.
Sqoop does not ship with third party JDBC drivers. You must download them separately and save them to the /var/lib/sqoop/ directory on the server.
Note:
The JDBC drivers need to be installed only on the machine where Sqoop runs. You do not need to install them on all hosts in your Hadoop cluster.
You can download driver from here : https://dev.mysql.com/downloads/connector/j/5.1.html
Try the exact command as like below.
sqoop import --connect "jdbc:mysql://localhost:3306/books"
--username=root --password=root --table authors --as-textfile --target-dir=/datasqoop/authors_db --columns "id, name, email" --split-by id --driver com.mysql.jdbc.Driver
This will resolve your issues.
Find the jar locations that are being used in sqoop, in my case, it is pointing to the link /usr/share/java/mysql-connector-java.jar
so when I check the link /usr/share/java/mysql-connector-java.jar it points to mysql-connector-java-5.1.17.jar
/usr/share/java/mysql-connector-java.jar -> mysql-connector-java-5.1.17.jar
as 5.1.17 is having this issue, try 5.1.37 or higher.
unlink /usr/share/java/mysql-connector-java.jar
ln -s /usr/share/java/mysql-connector-java.jar /usr/share/java/mysql-connector-java-5.1.37.jar