im trying to import a table from oracle to hive and i keep getting this error.
Im executing:
sqoop import -Dmapreduce.job.queuename=XXXXXX --connect
jdbc:oracle:XXX:#//XXXXXX --username XXXX --password-file=XXXX --query
"select descripcion,zona from base.test" --mapreduce-job-name
jobSqoop-test --target-dir /data/user/hive/warehouse/base.db/test
--split-by zona --map-column-java "ZONA=Double,DESCRIPCION=String" --delete-target-dir --as-parquetfile --compression-codec=snappy --null-string '\N' --null-non-string '\N' --num-mappers 1 --hive-import --hive-overwrite --hive-database base --hive-table test --direct
Error: java.lang.ClassCastException: java.lang.Double cannot be cast to java.nio.ByteBuffer
at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:338)
at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:271)
at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:187)
at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:161)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:123)
at org.apache.parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:179)
at org.apache.parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:46)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:670)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.sqoop.mapreduce.parquet.hadoop.HadoopParquetImportMapper.write(HadoopParquetImportMapper.java:61)
at org.apache.sqoop.mapreduce.ParquetImportMapper.map(ParquetImportMapper.java:72)
at org.apache.sqoop.mapreduce.ParquetImportMapper.map(ParquetImportMapper.java:38)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Any ideas?
Thnx
It fix using
Dsqoop.parquet.logical_types.decimal.enable=false
When I run the sqoop export command from the terminal, it works fine. But if I run the same command from the oozie workflow it is throwing me the following error.
ror: java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:122)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
Caused by: java.lang.RuntimeException: Can't parse input data: '2018-05-14,967,893,74,8863.330000000005,7617.07,1246.26'
at adjust_jazz_compare.__loadFromFields(adjust_jazz_compare.java:512)
at adjust_jazz_compare.parse(adjust_jazz_compare.java:430)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
... 10 more Caused by: java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:854)
at adjust_jazz_compare.__loadFromFields(adjust_jazz_compare.java:482)
... 12 more
Below is the command i am using
export --connect <jdbc> --username <user> --password <pass> --table <table> --export-dir <dir> --input-fields-terminated-by ',' --input-lines-terminated-by '\n'
Table properties in hive
hive.inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{field.delim=,, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=3973, numRows=70, rawDataSize=3903, COLUMN_STATS_ACCURATE=true, numFiles=1, transient_lastDdlTime=1530647041}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
The row which contains
"'2018-05-14,967,893,74,8863.330000000005,7617.07,1246.26'" is not able to parsed in sqoop.
From Hive terminal check the proper format of this row.May be some special character or some spaces are there because of that this data is not able to parsed.
Please share the Schema of table present in hive.And also your hive query.
Can you show how did you create the table? Maybe you forget about fields delimiter in Hive?
FIELDS TERMINATED BY ','
Example create table code:
CREATE TABLE IF NOT EXISTS employee (
eid int,
name String,
salary String,
destination String)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;
Getting error while running the sqoop import statement
sqoop import \
--connect jdbc:mysql://localhost:3306/retail_db \
--username retail_dba \
--password cloudera \
--table orders \
--target-dir /user/cloudera/problem1/orders \
--compress \
--compression-codec org.apache.hadoop.io.compress.SnappyCodec \
--as-avrodatafile;
Error message
17/11/06 21:56:30 WARN security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:ENOENT: No such file or directory
17/11/06 21:56:30 ERROR tool.ImportTool: Import failed: ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:716)
at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:498)
at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:479)
at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:495)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:616)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:203)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:176)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:273)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:127)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:513)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
When I am trying to import a table to Hive, I am getting a strange error.
Query:
sqoop import --connect 'jdbc:sybase:Tds:10.100.*.***:5500/DATABASE=****' --driver 'com.sybase.jdbc3.jdbc.SybDriver' --username "****" --password "***" --table dw.dm_court_courttype --direct -m 1 --hive-import --create-hive-table --hive-table DM_court_courtcype --target-dir "/user/hive/warehouse/DM_Court_CourtType" --verbose
Error:
java.io.IOException: SQLException in nextKeyValue
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:565)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: com.sybase.jdbc3.jdbc.SybSQLException: SQL Anywhere Error -131: Syntax error near '.' on line 1
at com.sybase.jdbc3.tds.Tds.a(Unknown Source)
at com.sybase.jdbc3.tds.Tds.nextResult(Unknown Source)
at com.sybase.jdbc3.tds.Tds.getResultSetResult(Unknown Source)
at com.sybase.jdbc3.tds.TdsCursor.open(Unknown Source)
at com.sybase.jdbc3.jdbc.SybStatement.executeQuery(Unknown Source)
at com.sybase.jdbc3.jdbc.SybPreparedStatement.executeQuery(Unknown Source)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
... 12 more
Don't use database name with table name.
Use --table dm_court_courttype instead of
--table dw.dm_court_courttype
Try this:
sqoop import --connect 'jdbc:sybase:Tds:10.100..:5500/DATABASE=****' --driver 'com.sybase.jdbc3.jdbc.SybDriver' --username "****" --password "*" --table dm_court_courttype --direct -m 1 --hive-import --create-hive-table --hive-table DM_court_courtcype --target-dir "/user/hive/warehouse/DM_Court_CourtType" --verbose
I am trying to use Sqoop to load files on HDFS into Oracle database:
Here is my input data:
Input data:
100|John|Miller|3.10
200|Sam|Madden|4.0
Here is the Sqoop command:
sqoop export --connect
"jdbc:oracle:thin:username/password#//host:port/service" --password
"pass" --username "user" --export-dir "/hdfs/path/"
--input-lines-terminated-by '\n' --input-null-string '\N' --input-null-non-string '\N' --input-fields-terminated-by '|' --table "SCRATCHPAD" --columns ID,FIRST_NAME,LAST_NAME,GPA
Here is the snippet of the error message that I see. Any help with this will be greatly appreciated.
INFO mapreduce.Job: Task Id :
attempt_1469238174088_466114_m_000001_1, Status : FAILED Error:
java.io.IOException: Can't export data, please check failed map task
logs at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1707)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused
by: java.lang.RuntimeException: Can't parse input data:
'200 Sam Madden 200.20'