Sqoop oracle connection issue - oracle

I'm trying to connect to oracle database to check the number of records.
Scenario-1:
[user#hostname ingestion]$ sqoop eval --connect jdbc:oracle:thin:#//hostname_1:PORT_1/Service_1 --username USER --password PASSWORD --query 'select count(*) from SCHEMA_1.TABLE_1'
class path is /usr/hdp/current/hive-client/lib/libthrift-0.9.3.jar:
Warning: /usr/hdp/2.5.3.0-37/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/05/19 14:50:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.3.0-37
17/05/19 14:50:25 INFO hdfs.PeerCache: SocketCache disabled.
17/05/19 14:50:26 INFO manager.SqlManager: Using default fetchSize of 1000
17/05/19 14:50:26 WARN tool.EvalSqlTool: SQL exception executing statement: java.sql.SQLException: Io exception: Oracle Error ORA-12650
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:327)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.tool.EvalSqlTool.run(EvalSqlTool.java:64)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
Scenario-2:
[user#hostname ingestion]$ sqoop eval --connect jdbc:oracle:thin:#//hostname_2:PORT_2/Service_2 --username USER --password PASSWORD --query 'select count(*) from SCHEMA_2.TABLE_2'
class path is /usr/hdp/current/hive-client/lib/libthrift-0.9.3.jar:
Warning: /usr/hdp/2.5.3.0-37/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/05/19 15:02:21 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.3.0-37
17/05/19 15:02:23 INFO hdfs.PeerCache: SocketCache disabled.
17/05/19 15:02:23 INFO manager.SqlManager: Using default fetchSize of 1000
17/05/19 15:02:24 INFO manager.OracleManager: Time zone has been set to GMT
------------------------
| COUNT(*) |
------------------------
| 43 |
------------------------
The first one is giving me error whereas the second one is giving me the expected result.
Can anyone help?
jar used ojdbc14-10.2.0.4.0.jar

After spending quite some time found the issue. The edge node where Sqoop is running cannot telnet because of the closed ports(OS Patching was done before week and the linux admins closed the ports and did not open them).
Once the ports are open in the edge node where sqoop runs I was able to execute the sqoop eval

Related

Sqoop's import-all-table is not working

Hi i am trying to import all table from all schema from Oracle DB to HDFS.
This is my script:
sqoop-import-all-tables -Dmapreduce.job.user.classpath.first=true -Dhadoop.security.credential.provider.path=jceks://x.jceks --connect jdbc:oracle:thin:#x.x.x.x:1521/yyyy --username xxxx --password xxxx --warehouse-dir /data-warehouse/xxxx --as-avrodatafile --compression-codec snappy --autoreset-to-one-mapper
When i am running this script, not getting any error and no any Job is starting.
Output:
Warning: /usr/hdp/2.6.2.0-205/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
find: failed to restore initial working directory: Permission denied
18/08/11 08:32:51 INFO sqoop.Sqoop: Running **Sqoop version: 1.4.6.2.6.2.0-205**
18/08/11 08:32:51 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/08/11 08:32:51 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
18/08/11 08:32:51 INFO manager.SqlManager: Using default fetchSize of 1000
18/08/11 08:32:53 INFO manager.OracleManager: Time zone has been set to IST
It seems that the user configured in sqoop does not have enough privileges to query and export the data from Oracle. Please check connect and query from command line to Oracle database.
Regards !!!

sqoop | getting nullpointerexception while running sqoop import tool

I am trying to import mysql table into hive using sqoop 1.4.7. Command I have executed
sqoop import --connect jdbc:mysql://localhost:3306/sb --username root --password xxxx --table sb_user --hive-import --create-hive-table --hive-table sqoop_import.test_from_sqoop --fields-terminated-by "," --bindir ./
But getting the below exception
Warning: /usr/local/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
18/04/03 22:44:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
18/04/03 22:44:56 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/04/03 22:44:56 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/04/03 22:44:56 INFO tool.CodeGenTool: Beginning code generation
18/04/03 22:44:57 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sb_user` AS t LIMIT 1
18/04/03 22:44:57 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sb_user` AS t LIMIT 1
18/04/03 22:44:57 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop
Note: ./sb_user.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/04/03 22:44:58 INFO orm.CompilationManager: Writing jar file: ./sb_user.jar
18/04/03 22:44:58 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
at java.util.Arrays.asList(Arrays.java:3800)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:76)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListing(FileListing.java:67)
at com.cloudera.sqoop.util.FileListing.getFileListing(FileListing.java:39)
at org.apache.sqoop.orm.CompilationManager.addClassFilesFromDir(CompilationManager.java:293)
at org.apache.sqoop.orm.CompilationManager.jar(CompilationManager.java:378)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:108)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
But I can execute the below command successfully
sqoop list-tables --connect jdbc:mysql://localhost:3306/information_schema --username root -password xxxx
Please help me here. Thanks
Problem is in --bindir ./ option. Sqoop tries to recursively add all contents inside ./ (https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/util/FileListing.java#L72). Check this dir or just change it.

Using Sqoop to import data from Redshift To Hive

I'm getting the error: Could not load db driver class.
The connection and error is below. Under that is a list of the jar files in the lib directory. What am i doing wrong?
sqoop import
--connect jdbc:redshift://< >
--username < > --password < >
--driver com.amazon.redshift.jdbc.Driver
--table import-all-tables
17/04/21 11:14:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
17/04/21 11:14:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/04/21 11:14:46 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
17/04/21 11:14:46 INFO manager.SqlManager: Using default fetchSize of 1000
17/04/21 11:14:46 INFO tool.CodeGenTool: Beginning code generation
17/04/21 11:14:46 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.amazon.redshift.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: com.amazon.redshift.jdbc.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:856)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:744)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1845)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
[t lib]$ ls
ant-contrib-1.0b3.jar hsqldb-1.8.0.10.jar kite-hadoop-compatibility-1.0.0.jar parquet-generator-1.4.1.jar
ant-eclipse-1.0-jvm1.2.jar jackson-annotations-2.3.0.jar mysql-connector-java.jar parquet-hadoop-1.4.1.jar
avro-1.7.5.jar jackson-core-2.3.1.jar opencsv-2.3.jar parquet-jackson-1.4.1.jar
avro-mapred-1.7.5-hadoop2.jar jackson-core-asl-1.9.13.jar paranamer-2.3.jar RedshiftJDBC42-1.2.1.1001 (2).jar
commons-codec-1.4.jar jackson-databind-2.3.1.jar parquet-avro-1.4.1.jar slf4j-api-1.6.1.jar
commons-compress-1.4.1.jar jackson-mapper-asl-1.9.13.jar parquet-column-1.4.1.jar snappy-java-1.0.5.jar
commons-io-1.4.jar kite-data-core-1.0.0.jar parquet-common-1.4.1.jar xz-1.0.jar
commons-jexl-2.1.1.jar kite-data-hive-1.0.0.jar parquet-encoding-1.4.1.jar
commons-logging-1.1.1.jar kite-data-mapreduce-1.0.0.jar parquet-format-2.0.0.jar
your jdbc is not exist in sqoop/lib so download your valid jdbc driver and copy to sqoop/lib

Sqoop export to sql server error

I want to export hdfs file to sql server. I'm using sqoop for that
sqoop export --bindir . --connect "jdbc:sqlserver://server;database=db" --username sa --password pwd --table sqoop_test -m 1 --export-dir /user/sqooptest
but i get the following error.
Warning: /usr/local/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
sqoop export --bindir . --connect "jdbc:sqlserver://server;database=db" --username sa --password pwd --table sqoop_test -m 1 --export-dir /user/sqooptest
Warning: /usr/local/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
16/07/30 03:59:06 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/07/30 03:59:07 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/07/30 03:59:07 INFO manager.SqlManager: Using default fetchSize of 1000
16/07/30 03:59:07 INFO tool.CodeGenTool: Beginning code generation
16/07/30 03:59:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM [sqoop_test] AS t WHERE 1=0
16/07/30 03:59:07 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop/hadoop-2.6.0
Note: ./sqoop_test.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/07/30 03:59:10 INFO orm.CompilationManager: Writing jar file: ./sqoop_test.jar
16/07/30 03:59:10 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
at java.util.Arrays.asList(Arrays.java:3800)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:76)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListing(FileListing.java:67)
at com.cloudera.sqoop.util.FileListing.getFileListing(FileListing.java:39)
at org.apache.sqoop.orm.CompilationManager.addClassFilesFromDir(CompilationManager.java:284)
at org.apache.sqoop.orm.CompilationManager.jar(CompilationManager.java:346)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:109)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
the file has only 3 rows with three columns each. it has no null values. I tried using --input-null-string as well.
my sql table :
create table sqoop_test
(id int,
name nvarchar(200),
title nvarchar(200))
and the file content in hdfs is,
5,X,analyst
6,Y,architect
7,Z,lead

error while using sqoop for data transfer to hdfs

i have used sqoop to transfer data between hdfs and oracle as shown below :
hadoop#jiogis-cluster-jiogis-master-001:~$ sqoop import --connect jdbc:oracle:gis-scan.ril.com/SAT --username=r4g_viewer --password=viewer_123 --table=R4G_OSP.ENODEB --hive-import --hive-table=ENODEB --target-dir=user/hive/warehouse/proddb/JioCenterBoundary -- direct
And i get error as shown below when i use sqoop as show above
Warning: /volumes/disk1/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /volumes/disk1/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /volumes/disk1/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/05/09 11:11:19 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/05/09 11:11:19 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/05/09 11:11:19 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
16/05/09 11:11:19 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
16/05/09 11:11:19 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
16/05/09 11:11:19 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: No manager for connect string: jdbc:oracle:gis-scan.ril.com/SAT
at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:191)
at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:256)
at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:89)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:593)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Your jdbc connection string does not look correct. Can you try it in this format:
--connect jdbc:oracle:thin:#//hostname:port/servicename
In your case, this is probably:
--connect jdbc:oracle:thin:#//gis-scan.ril.com:1521/SAT
You may want to double check the port number is correct as the scan listener may not be on the default 1521 port.

Resources