Sqoop Export Hive String to Oracle CLOB - oracle

I want to export STRING data from Hive to CLOB in Oracle.
Command :
sqoop export -Dsqoop.export.records.per.statement=1 --connect 'jdbc:oracle:thin:#192.168.41.67:1521:orcl' --username ILABUSER --password impetus --table ILABUSER.CDT_ORC_1 --export-dir /user/dev/db/123 --input-fields-terminated-by '\001' --input-null-string '\\N' --input-null-non-string '\\N' -m 2
Exception:
Error: java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: Could not buffer record
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:218)
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:84)
... 10 more
Caused by: java.lang.CloneNotSupportedException: com.cloudera.sqoop.lib.ClobRef
at java.lang.Object.clone(Native Method)
at org.apache.sqoop.lib.LobRef.clone(LobRef.java:109)
at ILABUSER_CDT_ORC_1.clone(ILABUSER_CDT_ORC_1.java:322)
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:213)
... 15 more

As a workaround I used --map-column-java tag.
I mapped Clob column (named col_clob) to String in Java.
Added below code in above command:
--map-column-java tag col_clob=String

Related

Sqoop: java.lang.Double cannot be cast to java.nio.ByteBuffer

im trying to import a table from oracle to hive and i keep getting this error.
Im executing:
sqoop import -Dmapreduce.job.queuename=XXXXXX --connect
jdbc:oracle:XXX:#//XXXXXX --username XXXX --password-file=XXXX --query
"select descripcion,zona from base.test" --mapreduce-job-name
jobSqoop-test --target-dir /data/user/hive/warehouse/base.db/test
--split-by zona --map-column-java "ZONA=Double,DESCRIPCION=String" --delete-target-dir --as-parquetfile --compression-codec=snappy --null-string '\N' --null-non-string '\N' --num-mappers 1 --hive-import --hive-overwrite --hive-database base --hive-table test --direct
Error: java.lang.ClassCastException: java.lang.Double cannot be cast to java.nio.ByteBuffer
at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:338)
at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:271)
at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:187)
at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:161)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:123)
at org.apache.parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:179)
at org.apache.parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:46)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:670)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.sqoop.mapreduce.parquet.hadoop.HadoopParquetImportMapper.write(HadoopParquetImportMapper.java:61)
at org.apache.sqoop.mapreduce.ParquetImportMapper.map(ParquetImportMapper.java:72)
at org.apache.sqoop.mapreduce.ParquetImportMapper.map(ParquetImportMapper.java:38)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Any ideas?
Thnx
It fix using
Dsqoop.parquet.logical_types.decimal.enable=false

Sqoop export is failing. Can't parse input data:'<data>'

When I run the sqoop export command from the terminal, it works fine. But if I run the same command from the oozie workflow it is throwing me the following error.
ror: java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:122)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
Caused by: java.lang.RuntimeException: Can't parse input data: '2018-05-14,967,893,74,8863.330000000005,7617.07,1246.26'
at adjust_jazz_compare.__loadFromFields(adjust_jazz_compare.java:512)
at adjust_jazz_compare.parse(adjust_jazz_compare.java:430)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
... 10 more Caused by: java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:854)
at adjust_jazz_compare.__loadFromFields(adjust_jazz_compare.java:482)
... 12 more
Below is the command i am using
export --connect <jdbc> --username <user> --password <pass> --table <table> --export-dir <dir> --input-fields-terminated-by ',' --input-lines-terminated-by '\n'
Table properties in hive
hive.inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{field.delim=,, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=3973, numRows=70, rawDataSize=3903, COLUMN_STATS_ACCURATE=true, numFiles=1, transient_lastDdlTime=1530647041}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
The row which contains
"'2018-05-14,967,893,74,8863.330000000005,7617.07,1246.26'" is not able to parsed in sqoop.
From Hive terminal check the proper format of this row.May be some special character or some spaces are there because of that this data is not able to parsed.
Please share the Schema of table present in hive.And also your hive query.
Can you show how did you create the table? Maybe you forget about fields delimiter in Hive?
FIELDS TERMINATED BY ','
Example create table code:
CREATE TABLE IF NOT EXISTS employee (
eid int,
name String,
salary String,
destination String)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;

Sqoop import error message ERROR tool.ImportTool: Import failed: ENOENT: No such file or directory Options

Getting error while running the sqoop import statement
sqoop import \
--connect jdbc:mysql://localhost:3306/retail_db \
--username retail_dba \
--password cloudera \
--table orders \
--target-dir /user/cloudera/problem1/orders \
--compress \
--compression-codec org.apache.hadoop.io.compress.SnappyCodec \
--as-avrodatafile;
Error message
17/11/06 21:56:30 WARN security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:ENOENT: No such file or directory
17/11/06 21:56:30 ERROR tool.ImportTool: Import failed: ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:716)
at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:498)
at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:479)
at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:495)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:616)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:203)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:176)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:273)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:127)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:513)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

Sqoop import from SybaseIQ to Hive - java.io.IOException: SQLException in nextKeyValue

When I am trying to import a table to Hive, I am getting a strange error.
Query:
sqoop import --connect 'jdbc:sybase:Tds:10.100.*.***:5500/DATABASE=****' --driver 'com.sybase.jdbc3.jdbc.SybDriver' --username "****" --password "***" --table dw.dm_court_courttype --direct -m 1 --hive-import --create-hive-table --hive-table DM_court_courtcype --target-dir "/user/hive/warehouse/DM_Court_CourtType" --verbose
Error:
java.io.IOException: SQLException in nextKeyValue
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:565)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: com.sybase.jdbc3.jdbc.SybSQLException: SQL Anywhere Error -131: Syntax error near '.' on line 1
at com.sybase.jdbc3.tds.Tds.a(Unknown Source)
at com.sybase.jdbc3.tds.Tds.nextResult(Unknown Source)
at com.sybase.jdbc3.tds.Tds.getResultSetResult(Unknown Source)
at com.sybase.jdbc3.tds.TdsCursor.open(Unknown Source)
at com.sybase.jdbc3.jdbc.SybStatement.executeQuery(Unknown Source)
at com.sybase.jdbc3.jdbc.SybPreparedStatement.executeQuery(Unknown Source)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
... 12 more
Don't use database name with table name.
Use --table dm_court_courttype instead of
--table dw.dm_court_courttype
Try this:
sqoop import --connect 'jdbc:sybase:Tds:10.100..:5500/DATABASE=****' --driver 'com.sybase.jdbc3.jdbc.SybDriver' --username "****" --password "*" --table dm_court_courttype --direct -m 1 --hive-import --create-hive-table --hive-table DM_court_courtcype --target-dir "/user/hive/warehouse/DM_Court_CourtType" --verbose

Sqoop export issue: parsing input data

I am trying to use Sqoop to load files on HDFS into Oracle database:
Here is my input data:
Input data:
100|John|Miller|3.10
200|Sam|Madden|4.0
Here is the Sqoop command:
sqoop export --connect
"jdbc:oracle:thin:username/password#//host:port/service" --password
"pass" --username "user" --export-dir "/hdfs/path/"
--input-lines-terminated-by '\n' --input-null-string '\N' --input-null-non-string '\N' --input-fields-terminated-by '|' --table "SCRATCHPAD" --columns ID,FIRST_NAME,LAST_NAME,GPA
Here is the snippet of the error message that I see. Any help with this will be greatly appreciated.
INFO mapreduce.Job: Task Id :
attempt_1469238174088_466114_m_000001_1, Status : FAILED Error:
java.io.IOException: Can't export data, please check failed map task
logs at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1707)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused
by: java.lang.RuntimeException: Can't parse input data:
'200 Sam Madden 200.20'

Resources