Importing data from Sybase IQ using Sqoop: Syntax error near 'Committed' on line 1 - sqoop

I am trying to import a table from Sybase IQ to a Hive table using the following sqoop command:
sqoop import \
--verbose \
--connect jdbc:sybase:Tds:server:port#?ServiceName=dbName \
--username username \
-P \
--driver com.sybase.jdbc4.jdbc.SybDriver \
--table myName.someTable \
--target-dir file:////tmp/HELLO/tej01
From this I get the following error thrown: Syntax error near 'Committed' on line 1
I've also tried escaping the "." in the table name using the section below but I get the same error:
--table myName\".\"someTable \
Can someone please give me some direction as to how I can resolve this?
19/12/05 01:45:34 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-mapr-1803
19/12/05 01:45:34 DEBUG tool.BaseSqoopTool: Enabled debug logging.
19/12/05 01:45:35 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
19/12/05 01:45:35 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
19/12/05 01:45:35 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
19/12/05 01:45:35 INFO manager.SqlManager: Using default fetchSize of 1000
19/12/05 01:45:35 INFO tool.CodeGenTool: Beginning code generation
19/12/05 01:45:35 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM myName.someTable AS t WHERE 1=0
19/12/05 01:45:35 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
19/12/05 01:45:35 ERROR manager.SqlManager: Error executing statement: com.sybase.jdbc4.jdbc.SybSQLException: SQL Anywhere Error -131: Syntax error near 'Committed' on line 1
com.sybase.jdbc4.jdbc.SybSQLException: SQL Anywhere Error -131: Syntax error near 'Committed' on line 1
at com.sybase.jdbc4.tds.Tds.processEed(Tds.java:4201)
at com.sybase.jdbc4.tds.Tds.nextResult(Tds.java:3318)
at com.sybase.jdbc4.jdbc.ResultGetter.nextResult(ResultGetter.java:78)
at com.sybase.jdbc4.jdbc.SybStatement.nextResult(SybStatement.java:302)
at com.sybase.jdbc4.jdbc.SybStatement.nextResult(SybStatement.java:284)
at com.sybase.jdbc4.jdbc.SybStatement.updateLoop(SybStatement.java:2762)
at com.sybase.jdbc4.jdbc.SybStatement.executeUpdate(SybStatement.java:2746)
at com.sybase.jdbc4.jdbc.SybPreparedStatement.executeUpdate(SybPreparedStatement.java:330)
at com.sybase.jdbc4.tds.Tds.setOption(Tds.java:2139)
at com.sybase.jdbc4.jdbc.SybConnection.setTransactionIsolation(SybConnection.java:2806)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:907)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:760)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:783)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:280)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:251)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:237)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:300)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1846)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1657)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:479)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:606)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
19/12/05 01:45:35 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1663)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:479)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:606)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

This seems to be a known bug (SAP KBA 2707539).
SAP Targeted CR
I was getting this error message with jconnect16.jar. However the error does not occur with jconn4.jar. Using version 4 fixed the issue for me.

Related

Sqoop error on data fetch from Oracle Database using Oracle wallet

sqoop import -D mapred.map.child.java.opts='-Doracle.net.tns_admin=. -Doracle.net.wallet_location=.' -files $HOME/wallet/cwallet.sso,$HOME/wallet/ewallet.p12,$HOME/wallet/sqlnet.ora,$HOME/wallet/tnsnames.ora --connect jdbc:oracle:thin:#TEST2 --table SQOOP_TEST
Am using this sqoop command to fetch data from Oracle DB but ending with below error
ERROR manager.SqlManager: Error executing statement: java.sql.SQLRecoverableException: IO Error: Unknown host specified
My Wallet:
tnsnames.ora
TEST2= (DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = <IP>)(PORT = 1521))
)
(CONNECT_DATA = (server=dedicated)
(SERVICE_NAME = HDPTEST )
) )
Wallet works fine,
$sqlplus /#TEST2
SQL*Plus: Release 11.2.0.4.0 Production on Thu Jun 8 14:48:42 2017
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release
11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>
JDBC without uding wallet works fine,
jdbc:oracle:thin:#:1521/HDPTEST
Detailed Log
Warning: /usr/local/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/local/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/06/08 15:06:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
17/06/08 15:06:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/06/08 15:06:16 DEBUG tool.BaseSqoopTool: Enabled debug logging.
17/06/08 15:06:16 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
17/06/08 15:06:16 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
17/06/08 15:06:16 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
17/06/08 15:06:16 DEBUG oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop can be called by Sqoop!
17/06/08 15:06:16 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
17/06/08 15:06:16 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
17/06/08 15:06:16 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:oracle:thin
17/06/08 15:06:16 DEBUG manager.OracleManager$ConnCache: Instantiated new connection cache.
17/06/08 15:06:16 INFO manager.SqlManager: Using default fetchSize of 1000
17/06/08 15:06:16 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.OracleManager#1d2adfbe
17/06/08 15:06:16 INFO tool.CodeGenTool: Beginning code generation
17/06/08 15:06:16 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM SQOOP_TEST t WHERE 1=0
17/06/08 15:06:16 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM SQOOP_TEST t WHERE 1=0
17/06/08 15:06:16 DEBUG manager.OracleManager: Creating a new connection for jdbc:oracle:thin:#TEST2, using username: null
17/06/08 15:06:16 DEBUG manager.OracleManager: No connection paramenters specified. Using regular API for making connection.
17/06/08 15:06:16 ERROR manager.SqlManager: Error executing statement: java.sql.SQLRecoverableException: IO Error: Unknown host specified
java.sql.SQLRecoverableException: IO Error: Unknown host specified
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:743)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:662)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:560)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:325)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:744)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: oracle.net.ns.NetException: Unknown host specified
at oracle.net.resolver.HostnameNamingAdapter.resolve(HostnameNamingAdapter.java:207)
at oracle.net.resolver.NameResolver.resolveName(NameResolver.java:131)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:476)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:595)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:230)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1452)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:496)
... 24 more
17/06/08 15:06:16 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Please help me out to find where am wrong.
Do you have oraclepki.jar, osdt_core.jar, and osdt_cert.jar in the classpath and also please check out the "SSL with JDBC" whitepaper for more details(page 9). Also, the Wallet location can be specified as part of the tnsnames.ora as shown here.
LISTENER = (ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcps)(HOST=servername)(PORT=2484))
) WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(
DIRECTORY=/server/wallet/path))
)

Using Sqoop to import data from Redshift To Hive

I'm getting the error: Could not load db driver class.
The connection and error is below. Under that is a list of the jar files in the lib directory. What am i doing wrong?
sqoop import
--connect jdbc:redshift://< >
--username < > --password < >
--driver com.amazon.redshift.jdbc.Driver
--table import-all-tables
17/04/21 11:14:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
17/04/21 11:14:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/04/21 11:14:46 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
17/04/21 11:14:46 INFO manager.SqlManager: Using default fetchSize of 1000
17/04/21 11:14:46 INFO tool.CodeGenTool: Beginning code generation
17/04/21 11:14:46 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.amazon.redshift.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: com.amazon.redshift.jdbc.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:856)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:744)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1845)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
[t lib]$ ls
ant-contrib-1.0b3.jar hsqldb-1.8.0.10.jar kite-hadoop-compatibility-1.0.0.jar parquet-generator-1.4.1.jar
ant-eclipse-1.0-jvm1.2.jar jackson-annotations-2.3.0.jar mysql-connector-java.jar parquet-hadoop-1.4.1.jar
avro-1.7.5.jar jackson-core-2.3.1.jar opencsv-2.3.jar parquet-jackson-1.4.1.jar
avro-mapred-1.7.5-hadoop2.jar jackson-core-asl-1.9.13.jar paranamer-2.3.jar RedshiftJDBC42-1.2.1.1001 (2).jar
commons-codec-1.4.jar jackson-databind-2.3.1.jar parquet-avro-1.4.1.jar slf4j-api-1.6.1.jar
commons-compress-1.4.1.jar jackson-mapper-asl-1.9.13.jar parquet-column-1.4.1.jar snappy-java-1.0.5.jar
commons-io-1.4.jar kite-data-core-1.0.0.jar parquet-common-1.4.1.jar xz-1.0.jar
commons-jexl-2.1.1.jar kite-data-hive-1.0.0.jar parquet-encoding-1.4.1.jar
commons-logging-1.1.1.jar kite-data-mapreduce-1.0.0.jar parquet-format-2.0.0.jar
your jdbc is not exist in sqoop/lib so download your valid jdbc driver and copy to sqoop/lib

sqoop null pointer exception

I am trying to run a sqoop import but it is failing with null pointer exception. Can somebody please help?
This is the command I ran:
sqoop import-all-tables --bindir ./ --num-mappers 1 --connect "jdbc:mysql://localhost/retail_db" --username=***** --password=****** --hive-import --hive-overwrite --create-hive-table --compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec --outdir java_files --bindir ./sqoop/
And this is what I get:
16/08/14 12:14:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/08/14 12:14:47 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/08/14 12:14:47 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
16/08/14 12:14:47 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
16/08/14 12:14:48 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
16/08/14 12:14:48 INFO tool.CodeGenTool: Beginning code generation
16/08/14 12:14:48 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
16/08/14 12:14:48 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
16/08/14 12:14:48 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop
Note: ./categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/08/14 12:14:49 INFO orm.CompilationManager: Writing jar file: ./categories.jar
16/08/14 12:14:49 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
java.lang.NullPointerException
at java.util.Arrays$ArrayList.<init>(Arrays.java:2842)
at java.util.Arrays.asList(Arrays.java:2828)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:76)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListingNoSort(FileListing.java:82)
at org.apache.sqoop.util.FileListing.getFileListing(FileListing.java:67)
at com.cloudera.sqoop.util.FileListing.getFileListing(FileListing.java:39)
at org.apache.sqoop.orm.CompilationManager.addClassFilesFromDir(CompilationManager.java:284)
at org.apache.sqoop.orm.CompilationManager.jar(CompilationManager.java:346)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:109)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:111)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Sqoop create-hive-table SQL Server NON-DEFAULT schema

HDP-2.4.2.0-258 installed using Ambari 2.2.2.0
In SQL Server :
TABLE_CATALOG TABLE_SCHEMA TABLE_NAME
Management Administration SettingAttribute
Management Administration SettingAttributeGroup
Management Administration SettingAttributeValue
Management Administration SettingValue
Management ape DatabaseScriptLog
Management ape DatabaseLog
Management Common Language
Management Common ThirdPartyType
Management Common Country
Management Company DistributorCow
Management Company CustomerSetting
Management Company CustomerSettingAttributeValue
I can list the databases and the tables in a schema :
-bash-4.2$ sqoop list-databases --connect 'jdbc:sqlserver://<IP>;database=FleetManagement' --username --password
find: failed to restore initial working directory: Permission denied
16/08/11 11:25:39 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
16/08/11 11:25:39 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/08/11 11:25:39 INFO manager.SqlManager: Using default fetchSize of 1000
master
tempdb
model
msdb
Auth
FeatureToggle
FleetManagementCoach
LatestRuntime
FleetManagementThirdParty
VehicleDriverServicesFollowUp
FleetManagementCustomer
FleetManagementMessaging
FleetManagementSubscription
FleetManagementSupport
FleetManagementFollowUp
FleetManagementDatawarehouse
FleetManagement
FleetManagementPositioning
-bash-4.2$ sqoop list-tables --connect 'jdbc:sqlserver://<IP>;database=FleetManagement' --username --password -- --schema Administration
find: failed to restore initial working directory: Permission denied
16/08/11 11:25:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
16/08/11 11:25:12 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/08/11 11:25:12 INFO manager.SqlManager: Using default fetchSize of 1000
16/08/11 11:25:12 INFO manager.SQLServerManager: We will use schema Administration
SettingAttribute
SettingAttributeGroup
SettingAttributeValue
SettingValue
Now, while using create-hive-table, Sqoop is unable to create SettingAttribute table
I tried the following commands in vain :
sqoop create-hive-table --driver 'com.microsoft.sqlserver.jdbc.SQLServerDriver' --connection-manager org.apache.sqoop.manager.SQLServerManager --connect 'jdbc:sqlserver://<IP>;database=FleetManagement' --username --password --table 'Administration.SettingAttribute'
The output :
16/08/10 16:40:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
16/08/10 16:40:32 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/08/10 16:40:32 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
16/08/10 16:40:32 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
16/08/10 16:40:32 INFO manager.SqlManager: Using default fetchSize of 1000
16/08/10 16:40:33 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM [Administration.SettingAttribute] AS t WHERE 1=0
16/08/10 16:40:33 ERROR manager.SqlManager: Error executing statement: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'Administration.SettingAttribute'.
com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'Administration.SettingAttribute'.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1655)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:440)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:385)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:297)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:758)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
at org.apache.sqoop.hive.TableDefWriter.getCreateTableStmt(TableDefWriter.java:126)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:188)
at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/08/10 16:40:34 INFO hive.HiveImport: Loading uploaded data into Hive
16/08/10 16:40:34 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist
Logging initialized using configuration in jar:file:/usr/hdp/2.4.2.0-258/hive/lib/hive-common-1.2.1000.2.4.2.0-258.jar!/hive-log4j.properties
NoViableAltException(307#[])
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:11578)
at org.apache.hadoop.hive.ql.parse.HiveParser.identifier(HiveParser.java:45881)
at org.apache.hadoop.hive.ql.parse.HiveParser.columnNameType(HiveParser.java:38052)
at org.apache.hadoop.hive.ql.parse.HiveParser.columnNameTypeList(HiveParser.java:36183)
at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:5222)
at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2648)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1658)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1117)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:316)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1189)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1126)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:314)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:412)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:428)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:717)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:338)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:245)
at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
FAILED: ParseException line 1:63 cannot recognize input near ')' 'COMMENT' ''Imported by sqoop on 2016/08/10 16:40:33'' in column specification
Even this one failed :
sqoop create-hive-table --driver 'com.microsoft.sqlserver.jdbc.SQLServerDriver' --connection-manager org.apache.sqoop.manager.SQLServerManager --connect 'jdbc:sqlserver://<IP>;database=FleetManagement' -- --schema Administration --table 'SettingAttribute' --username --password
The output :
16/08/10 16:42:37 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
16/08/10 16:42:37 ERROR tool.BaseSqoopTool: Error parsing arguments for create-hive-table:
16/08/10 16:42:37 ERROR tool.BaseSqoopTool: Unrecognized argument: --
16/08/10 16:42:37 ERROR tool.BaseSqoopTool: Unrecognized argument: --schema
16/08/10 16:42:37 ERROR tool.BaseSqoopTool: Unrecognized argument: Administration
16/08/10 16:42:37 ERROR tool.BaseSqoopTool: Unrecognized argument: --table
16/08/10 16:42:37 ERROR tool.BaseSqoopTool: Unrecognized argument: SettingAttribute
16/08/10 16:42:37 ERROR tool.BaseSqoopTool: Unrecognized argument: --username
Doesn't the create-hive-table support --schema option(nothing mentioned in the documentation)
use.. "-- --schema" at the end of the statement.
sqoop create-hive-table --driver 'com.microsoft.sqlserver.jdbc.SQLServerDriver' --connection-manager org.apache.sqoop.manager.SQLServerManager --connect 'jdbc:sqlserver://;database=FleetManagement' --table 'SettingAttribute' --username --password -- --schema Administration

How to run Sqoop with custom JDBC Driver?

I can run Sqoop without providing --driver parameter if I supply (--connect/--user/--password) for oracle thin.
But I need to get it running with custom JDBC driver (it properly implements java.sql.Driver interface) used in my project instead of oracle.jdbc.OracleDriver.
I wasn't able to get it working by simply proving it with --driver parameter.
And this suggestion wasn't helpful at all.
How to use Sqoop with custom DB access drivers?
How to overcome errors that I get?
If it has something to do with connection managers, could someone tell me what connection manager should I specify?
Thank you!
Here is what I'm actually trying to do:
./sqoop.sh import \
--fs $HDFS --jt $JT \
--connect <cutom-connection-string> --username username --password password \
--table SYS.ALL_TABLES --split-by TABLE_NAME --target-dir /temp/try/110 --verbose \
--driver xx.xx.xx.MyDriver
I get an error:
ERROR manager.SqlManager: Error executing statement:
java.sql.SQLException: ORA-00933: SQL command not properly ended
More error info:
DEBUG tool.BaseSqoopTool: Enabled debug logging.
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
INFO manager.SqlManager: Using default fetchSize of 1000
INFO tool.CodeGenTool: Beginning code generation
INFO xx.xx.xx.MyDriver: xx.xx.xx.MyDriver registered successfully.
DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
INFO xx.xx.xx.MyDriver: Returning database connection
DEBUG manager.SqlManager: Using fetchSize for next query: 1000
INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM SYS.ALL_TABLES AS t WHERE 1=0
ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: ORA-00933: SQL command not properly ended
java.sql.SQLException: ORA-00933: SQL command not properly ended
Your custom JDBC driver is used correctly. The problem seems to be in Generic jdbc connector that is in use and that seems to be producing invalid query. You might need to also fork the build in Oracle connector by replacing the driver with your custom one.
Jarcec

Resources