I'm trying to import some records from source using sqoop. I want to add all the output (from multiple runs) to the same folder & also have custom basename for each run (using -Dmapreduce.output.basename).
Files gets successfully created in the TEMPORARY directory, but for some reason when it is trying to move to the actual HDFS output location, it says "ignored"..
Can someone please advise what is causing it to be ignored. Below is the sqoop command with the output (used verbose). sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.12.0.
Also note that the target HDFS directory was empty when I tried this.
user_1#system1 /home/sriadmin$ sqoop-import -Dmapreduce.output.basename="12345_" --connect "jdbc:sap://xx.xx.xx.xx:30215" --username "EDHQUERY" -password xxxxxxxxxx --driver "com.sap.db.jdbc.Driver" --query "SELECT PLANT FROM SAPBWT.PLANT where \$CONDITIONS" --split-by "PLANT" --target-dir "/tmp/SRITEST/HANA_Extract_Plant/" --as-textfile --fields-terminated-by "|" --num-mappers "2" --append --verbose
Map-Reduce Framework
Map input records=6428
Map output records=6428
Input split bytes=212
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=305
CPU time spent (ms)=4760
Physical memory (bytes) snapshot=737996800
Virtual memory (bytes) snapshot=10986500096
Total committed heap usage (bytes)=1584398336
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=31774
18/05/29 11:19:30 INFO mapreduce.ImportJobBase: Transferred 31.0293 KB in 41.4266 seconds (766.9945 bytes/sec)
18/05/29 11:19:30 INFO mapreduce.ImportJobBase: Retrieved 6428 records.
18/05/29 11:19:30 DEBUG util.ClassLoaderStack: Restoring classloader: sun.misc.Launcher$AppClassLoader#47d384ee
18/05/29 11:19:30 INFO util.AppendUtils: Appending to directory HANA_Extract_Plant
18/05/29 11:19:30 DEBUG util.AppendUtils: Filename: 12345_-m-00000 ignored
18/05/29 11:19:30 DEBUG util.AppendUtils: Filename: 12345_-m-00001 ignored
18/05/29 11:19:30 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
18/05/29 11:19:30 DEBUG util.AppendUtils: Deleting temporary folder 634e75fe8ca94ba79c0b5fbbffdffe18_4056a33f
This is an existing bug -
https://issues.apache.org/jira/browse/SQOOP-3330
Sqoop moves only files with filename starting with 'part*'. .. The work around is to maintain your -Dmapreduce.output.basename="part.*"
(example : -in my case Dmapreduce.output.basename="part.12345_" )
Related
I'm trying to export a table from HDFS to Oracle database, using the command:
sqoop export --connect jdbc:oracle:thin:#ip:port/db --username user -P --table OPORTUNIDADESHIVE --export-dir /user/hadoop/OPORTUNIDADES/000000_0 --input-fields-terminated-by "\t"
where OPORTUNIDADESHIVE is the table from Oracle and the file "000000_0" is the table extracted from Hive to HDFS. Both tables have the same columns.
This table from Hive is ROW FORMAT DELIMITED and FIELDS TERMINATED BY '\t'.
But in the end, the export gives me this error message:
2021-11-04 14:58:04,633 INFO mapreduce.Job: map 0% reduce 0%
2021-11-04 14:58:13,711 INFO mapreduce.Job: map 100% reduce 0%
2021-11-04 14:58:13,723 INFO mapreduce.Job: Job job_1635324128846_0049
failed with state FAILED due to: Task failed
task_1635324128846_0049_m_000000 Job failed as tasks failed.
failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
2021-11-04 14:58:13,792 INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=2
Killed map tasks=2
Launched map tasks=2
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=28818
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=14409
Total vcore-milliseconds taken by all map tasks=14409
Total megabyte-milliseconds taken by all map tasks=29509632 2021-11-04 14:58:13,799 WARN mapreduce.Counters: Group
FileSystemCounters is deprecated. Use
org.apache.hadoop.mapreduce.FileSystemCounter instead 2021-11-04
14:58:13,800 INFO mapreduce.ExportJobBase: Transferred 0 bytes in
19.1507 seconds (0 bytes/sec) 2021-11-04 14:58:13,803 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is
deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2021-11-04 14:58:13,804 INFO mapreduce.ExportJobBase: Exported 0
records. 2021-11-04 14:58:13,804 ERROR mapreduce.ExportJobBase: Export
job failed! 2021-11-04 14:58:13,804 ERROR tool.ExportTool: Error
during export: Export job failed!
at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445)
at org.apache.sqoop.manager.OracleManager.exportTable(OracleManager.java:465)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
sqoop export is very sensitive to data type, length mismatch issue. so plese follow below steps to fix this issue -
Put hive and oracle definition side by side and check if data type and length are comparable. please note some data type like string in hive can hold more than 4000 char so you may need to handle such scenario specially.
if you see above comparison is fine then use this command to import data. Your command has export dir argument, it should be path to the table which is /user/hadoop/OPORTUNIDADES.
But i feel like its easy to use --hcatalog-table and here is complete command.
sqoop export --connect jdbc:oracle:thin:#mydb:5217/user --username user --password pass --hcatalog-database hive_db --hcatalog-table hive_tab --table orac_tab --input-null-string '\\\\N' --input-null-non-string '\\\\N' --m 6
Pls note we are using just input and output table and optional null handling.
when importing the data from oracle database to HDFS using Apache sqoop. it's imported but empty files.
sqoop import --connect jdbc:oracle:thin:#192.168.0.15:1521:XE --username system --password system --table EMP -m 1 --target-dir /user/sinha
after running its creates part-m-00000 file without any data...while running query
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/03/05 09:43:57 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.12.0
18/03/05 09:43:57 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/03/05 09:44:00 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
18/03/05 09:44:58 INFO mapreduce.JobSubmitter: number of splits:1
18/03/05 09:45:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1520229051986_0016
18/03/05 09:45:03 INFO impl.YarnClientImpl: Submitted application application_1520229051986_0016
18/03/05 09:45:03 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1520229051986_0016/
18/03/05 09:45:03 INFO mapreduce.Job: Running job: job_1520229051986_0016
18/03/05 09:45:54 INFO mapreduce.Job: Job job_1520229051986_0016 running in uber mode : false
18/03/05 09:45:54 INFO mapreduce.Job: map 0% reduce 0%
18/03/05 09:46:35 INFO mapreduce.Job: map 100% reduce 0%
18/03/05 09:46:36 INFO mapreduce.Job: Job job_1520229051986_0016 completed successfully
18/03/05 09:46:36 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=151209
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=0
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=37383
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=37383
Total vcore-milliseconds taken by all map tasks=37383
Total megabyte-milliseconds taken by all map tasks=38280192
Map-Reduce Framework
Map input records=0
Map output records=0
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=546
CPU time spent (ms)=5110
Physical memory (bytes) snapshot=143175680
Virtual memory (bytes) snapshot=1509150720
Total committed heap usage (bytes)=74973184
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
18/03/05 09:46:36 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 108.9264 seconds (0 bytes/sec)
18/03/05 09:46:36 INFO mapreduce.ImportJobBase: Retrieved 0 records
i don't know what is the problem?
Even if i check with "eval" command also it's display only column names of the table.
Looking at the logs, you source table don't have any records at all. Do a select * on your oracle table to validate. Add some records to your oracle table and try the sqoop operation again. You shall be able to fetch the data.
HDP-2.4.2.0-258 installed using Ambari 2.2.2.0
I have to import several SQL Server schema which should be accessible via Hive, Pig, MR and any third party(in future). I decided to import in HCatalog.
Sqoop provides ways to import to Hive OR HCatalog, I guess if I import to HCatalog, the same table will be accessible from Hive CLI, to MR and to Pig(please evaluate my assumption).
Questions :
If imported to Hive directly, will the table be available to Pig, MR
?
If imported to HCatalog, what needs to be done for accessing via Hive ?
Is pre-creation of tables in Hive necessary ? If yes, what is the advantage of the importing in HCatalog, (I can import in Hive directly)/(import in HDFS and then create external table) ?
Issue :
I wish to achieve the following in one step :
Import of the data(from SQL Server tables)
Avoid 'pre-creating' or writing create statements for those tables(there are 100s of them)
Store the table in ORC format
Store this data at custom HDFS path say /org/data/schema1, /org/data/schema2 and so on(is this impossible because Sqoop says it (--target-dir/--warehouse-dir)
I executed the following command :
-bash-4.2$ sqoop import --connect 'jdbc:sqlserver://<IP>;database=FleetManagement' --username --password --table SettingAttribute -- --schema Administration --hcatalog-home /usr/hdp/current/hive-webhcat --hcatalog-database default --hcatalog-table SettingAttribute --create-hcatalog-table --hcatalog-storage-stanza "stored as orcfile"
The source table contains 109 records and those are fetched :
16/08/10 15:02:27 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
16/08/10 15:02:27 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/08/10 15:02:28 INFO manager.SqlManager: Using default fetchSize of 1000
16/08/10 15:02:28 INFO manager.SQLServerManager: We will use schema Administration
16/08/10 15:02:28 INFO tool.CodeGenTool: Beginning code generation
16/08/10 15:02:28 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM [Administration].[SettingAttribute] AS t WHERE 1=0
16/08/10 15:02:28 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.4.2.0-258/hadoop-mapreduce
Note: /tmp/sqoop-ojoqcu/compile/dfab14748c41a566ec286b7e4b11004d/SettingAttribute.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/08/10 15:02:30 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ojoqcu/compile/dfab14748c41a566ec286b7e4b11004d/SettingAttribute.jar
16/08/10 15:02:30 INFO mapreduce.ImportJobBase: Beginning import of SettingAttribute
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/08/10 15:02:31 INFO impl.TimelineClientImpl: Timeline service address: http://l4373t.sss.com:8188/ws/v1/timeline/
16/08/10 15:02:31 INFO client.RMProxy: Connecting to ResourceManager at l4283t.sss.com/138.106.9.80:8050
16/08/10 15:02:33 INFO db.DBInputFormat: Using read commited transaction isolation
16/08/10 15:02:33 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN([SettingAttributeId]), MAX([SettingAttributeId]) FROM [Administration].[SettingAttribute]
16/08/10 15:02:33 INFO mapreduce.JobSubmitter: number of splits:4
16/08/10 15:02:33 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1467787344827_0013
16/08/10 15:02:34 INFO impl.YarnClientImpl: Submitted application application_1467787344827_0013
16/08/10 15:02:34 INFO mapreduce.Job: The url to track the job: http://l4283t.sss.com:8088/proxy/application_1467787344827_0013/
16/08/10 15:02:34 INFO mapreduce.Job: Running job: job_1467787344827_0013
16/08/10 15:02:41 INFO mapreduce.Job: Job job_1467787344827_0013 running in uber mode : false
16/08/10 15:02:41 INFO mapreduce.Job: map 0% reduce 0%
16/08/10 15:02:47 INFO mapreduce.Job: map 100% reduce 0%
16/08/10 15:02:48 INFO mapreduce.Job: Job job_1467787344827_0013 completed successfully
16/08/10 15:02:48 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=616636
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=540
HDFS: Number of bytes written=10079
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=16132
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=16132
Total vcore-seconds taken by all map tasks=16132
Total megabyte-seconds taken by all map tasks=66076672
Map-Reduce Framework
Map input records=109
Map output records=109
Input split bytes=540
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=320
CPU time spent (ms)=6340
Physical memory (bytes) snapshot=999870464
Virtual memory (bytes) snapshot=21872697344
Total committed heap usage (bytes)=943194112
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=10079
16/08/10 15:02:48 INFO mapreduce.ImportJobBase: Transferred 9.8428 KB in 17.2115 seconds (585.597 bytes/sec)
16/08/10 15:02:48 INFO mapreduce.ImportJobBase: Retrieved 109 records.
The files are created under my user :
hdfs dfs -ls /user/ojoqcu/SettingAttribute
Found 5 items
-rw------- 3 ojoqcu hdfs 0 2016-08-10 15:02 /user/ojoqcu/SettingAttribute/_SUCCESS
-rw------- 3 ojoqcu hdfs 8378 2016-08-10 15:02 /user/ojoqcu/SettingAttribute/part-m-00000
-rw------- 3 ojoqcu hdfs 144 2016-08-10 15:02 /user/ojoqcu/SettingAttribute/part-m-00001
-rw------- 3 ojoqcu hdfs 1123 2016-08-10 15:02 /user/ojoqcu/SettingAttribute/part-m-00002
-rw------- 3 ojoqcu hdfs 434 2016-08-10 15:02 /user/ojoqcu/SettingAttribute/part-m-00003
I cannot see anything in HCatalog(nor in Hive)
-bash-4.2$ /usr/hdp/2.4.2.0-258/hive-hcatalog/bin/hcat -e "show tables in default;"
WARNING: Use "yarn jar" to launch YARN applications.
16/08/10 15:07:12 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist
OK
Time taken: 2.007 seconds
Is there some authorization issue ?
I checked the var/log but none exist for Sqoop, Hive-Hcatalog and Hive, how can I view the authorization issue and fix it ?
Well, I'm unsure whether it was an authorization issue or a mere parsing problem or both. I did the following and it worked :
Did an su hive
Executed the following command(probably, the -- --schema should be
the last arg, Sqoop simply ignores/breaks after that!)
sqoop import --hcatalog-home /usr/hdp/current/hive-webhcat --hcatalog-database FleetManagement_Ape --hcatalog-table DatabaseLog --create-hcatalog-table --hcatalog-storage-stanza "stored as orcfile" --connect 'jdbc:sqlserver://<IP>;database=FleetManagement' --username --password --table DatabaseLog -- --schema ape
I am new to HDFS and I am trying to import data from my oracle 12c db. I have a table EMP, it needs to be imported inside hdfs as well as hive tables.
My data is getting created inside hdfs ('/user/hdfs' a folder 'EMP' gets created). But when I open hive query editor and type 'show tables' I don't get to see any tables here. I need the tables to be created inside HIVE as well
I am running the following commands.
1. Since am running sqoop as root user
usermod -a -G supergroup hardik
2.
export SQOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop
export HIVE_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive
export HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ojdbc7.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive/lib/*
export HADOOP_USER_NAME=hdfs
3.
export PATH=$PATH:$HIVE_HOME/bin
Now am running the SQOOP import command and I get the following on the console
4.
sqoop import --connect jdbc:oracle:thin:#bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP --hive-import -m 1 --create-hive-table --hive-table EMP
[root#bigdatadev1 ~]# sqoop import --connect jdbc:oracle:thin:#bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP --hive-import -m 1 --create-hive-table --hive-table EMP
Warning: /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/04/07 22:15:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.1
16/04/07 22:15:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/04/07 22:15:23 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
16/04/07 22:15:23 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
16/04/07 22:15:23 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
16/04/07 22:15:23 INFO manager.SqlManager: Using default fetchSize of 1000
16/04/07 22:15:23 INFO tool.CodeGenTool: Beginning code generation
16/04/07 22:15:24 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:15:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EMP t WHERE 1=0
16/04/07 22:15:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
Note: /tmp/sqoop-root/compile/fcb6484db042a7b4295d911956145a4e/EMP.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/04/07 22:15:25 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/fcb6484db042a7b4295d911956145a4e/EMP.jar
16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:15:25 INFO mapreduce.ImportJobBase: Beginning import of EMP
16/04/07 22:15:25 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:15:26 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
16/04/07 22:15:26 INFO client.RMProxy: Connecting to ResourceManager at bigdata/10.103.25.39:8032
16/04/07 22:15:30 INFO db.DBInputFormat: Using read commited transaction isolation
16/04/07 22:15:30 INFO mapreduce.JobSubmitter: number of splits:1
16/04/07 22:15:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460040138373_0007
16/04/07 22:15:31 INFO impl.YarnClientImpl: Submitted application application_1460040138373_0007
16/04/07 22:15:31 INFO mapreduce.Job: The url to track the job: http://bigdata:8088/proxy/application_1460040138373_0007/
16/04/07 22:15:31 INFO mapreduce.Job: Running job: job_1460040138373_0007
16/04/07 22:15:37 INFO mapreduce.Job: Job job_1460040138373_0007 running in uber mode : false
16/04/07 22:15:37 INFO mapreduce.Job: map 0% reduce 0%
16/04/07 22:15:43 INFO mapreduce.Job: Task Id : attempt_1460040138373_0007_m_000000_0, Status : FAILED
Error: EMP : Unsupported major.minor version 52.0
16/04/07 22:15:56 INFO mapreduce.Job: Task Id : attempt_1460040138373_0007_m_000000_1, Status : FAILED
Error: EMP : Unsupported major.minor version 52.0
16/04/07 22:16:03 INFO mapreduce.Job: map 100% reduce 0%
16/04/07 22:16:04 INFO mapreduce.Job: Job job_1460040138373_0007 completed successfully
16/04/07 22:16:04 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=137942
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=12
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Failed map tasks=2
Launched map tasks=3
Other local map tasks=3
Total time spent by all maps in occupied slots (ms)=20742
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=20742
Total vcore-seconds taken by all map tasks=20742
Total megabyte-seconds taken by all map tasks=10619904
Map-Reduce Framework
Map input records=3
Map output records=3
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=53
CPU time spent (ms)=2090
Physical memory (bytes) snapshot=207478784
Virtual memory (bytes) snapshot=2169630720
Total committed heap usage (bytes)=134217728
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=12
16/04/07 22:16:04 INFO mapreduce.ImportJobBase: Transferred 12 bytes in 38.6207 seconds (0.3107 bytes/sec)
16/04/07 22:16:04 INFO mapreduce.ImportJobBase: Retrieved 3 records.
16/04/07 22:16:05 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:16:05 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EMP t WHERE 1=0
16/04/07 22:16:05 INFO hive.HiveImport: Loading uploaded data into Hive
Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/hive-common-1.1.0-cdh5.5.1.jar!/hive-log4j.properties
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. AlreadyExistsException(message:Table EMP already exists)
I have tried all variations of sqoop import commands but none have succeeded. I am even more confuse today. Please help. Please do not mark this as duplicate.
From your logs, I found two errors:
Error: EMP : Unsupported major.minor version 52.0
Unsupported major.minor version 52.0 comes when you are trying to run a class compiled using Java 1.8 compiler into a lower JRE version e.g. JRE 1.7 or JRE 1.6. Check more here.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. AlreadyExistsException(message:Table EMP already exists)
Your job worked till taking data to hdfs. You must be trying same command again without deleting this /user/hdfs/EMP directory. That's why you got this error.
Check this related answer.
I have a table in Oracle XE 11g
SQL> create table bloblkup (
2 id NUMBER PRIMARY KEY,
3 name varchar(28) NOT NULL,
4 fdata BLOB
5 );
Table created.
SQL> desc bloblkup
Name Null? Type
----------------------------------------- -------- ----------------------------
ID NOT NULL NUMBER
NAME NOT NULL VARCHAR2(28)
FDATA BLOB
populated with
SQL> select * from bloblkup;
ID NAME
---------- ----------------------------
FDATA
--------------------------------------------------------------------------------
1 photo.jpg
032135435135
From the cluster I attempt to Sqoop this table into HDFS
sqoop import --connect jdbc:oracle:thin:#Rhea:1521:xe --username SYSTEM --password oracle --table bloblkup --columns 'name' -m 1
that executes to completion every time but provides INFO
15/03/24 09:14:39 INFO mapreduce.ImportJobBase: Retrieved 0 records.
I can retrieve databases and tables.
I am logging in as system which created and owns the table.
I have also found that I can query tables such as the all_tables and return rows just not table that I have created through sqlplus>
I added --m 1 flag after the first attempted failed due to being unable to located an primary key for the table. I added a primary key to the table using alter table with no change.
thoughts?
console output:
[root#sandbox ~]# sqoop import --connect jdbc:oracle:thin:#Rhea:1521:xe --username SYSTEM --password oracle --table bloblkup --columns 'name' -m 1
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/03/24 09:14:02 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.2.1.1.0-385
15/03/24 09:14:02 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/03/24 09:14:02 INFO manager.SqlManager: Using default fetchSize of 1000
15/03/24 09:14:02 INFO tool.CodeGenTool: Beginning code generation
15/03/24 09:14:04 INFO manager.OracleManager: Time zone has been set to GMT
15/03/24 09:14:05 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM bloblkup t WHERE 1=0
15/03/24 09:14:05 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-root/compile/ce267f99c7e1b14da474c2c395368b67/bloblkup.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
15/03/24 09:14:08 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/ce267f99c7e1b14da474c2c395368b67/bloblkup.jar
15/03/24 09:14:08 INFO manager.OracleManager: Time zone has been set to GMT
15/03/24 09:14:08 INFO mapreduce.ImportJobBase: Beginning import of bloblkup
15/03/24 09:14:09 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/03/24 09:14:10 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/03/24 09:14:10 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/192.168.1.91:8050
15/03/24 09:14:12 INFO db.DBInputFormat: Using read commited transaction isolation
15/03/24 09:14:13 INFO mapreduce.JobSubmitter: number of splits:1
15/03/24 09:14:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1427151026592_0037
15/03/24 09:14:14 INFO impl.YarnClientImpl: Submitted application application_1427151026592_0037
15/03/24 09:14:14 INFO mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1427151026592_0037/
15/03/24 09:14:14 INFO mapreduce.Job: Running job: job_1427151026592_0037
15/03/24 09:14:27 INFO mapreduce.Job: Job job_1427151026592_0037 running in uber mode : false
15/03/24 09:14:27 INFO mapreduce.Job: map 0% reduce 0%
15/03/24 09:14:38 INFO mapreduce.Job: map 100% reduce 0%
15/03/24 09:14:39 INFO mapreduce.Job: Job job_1427151026592_0037 completed successfully
15/03/24 09:14:39 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=107031
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=0
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=8553
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=8553
Total vcore-seconds taken by all map tasks=8553
Total megabyte-seconds taken by all map tasks=2138250
Map-Reduce Framework
Map input records=0
Map output records=0
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=76
CPU time spent (ms)=2170
Physical memory (bytes) snapshot=145907712
Virtual memory (bytes) snapshot=897458176
Total committed heap usage (bytes)=75497472
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
15/03/24 09:14:39 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 28.8478 seconds (0 bytes/sec)
15/03/24 09:14:39 INFO mapreduce.ImportJobBase: Retrieved 0 records.
Update the Oracle driver version ojdbc6.jar appears to work, but be flaky with JDK 1.7 use ojdbc7.jar
Also, you must "commit" database changes in SQLPLUS for them to persist
Check whether jdbc driver is under $SQOOP_HOME/lib if not copy the ojdbc6.jar file to:
/usr/lib/sqoop/lib/ directory
Provide more details from console.
If every thing is fine then add --target-dir to see output on that specific directory.
/usr/bin/sqoop import --connect jdbc:oracle:thin:system/system#<IP address>:1521:xe --username <username> -P--table <database name>.<table name> --columns "<column names>" --target-dir <target directory path> -m 1