i'm currently running a single node yarn cluster, and for some reason, i can't execute even a example that comes with map reduce (grep, wordcount, etc). With this line i execute grep:
$HADOOP_HOME/bin/yarn jar /usr/local/hadoop/share/hadoop/yarn/lib/hadoop-mapreduce-examples-2.4.0.jar grep input output2 'dfs[a-z.]+'
This cluster was previosly running Giraph programs, but rigth now i need a Map Reduce application, so i switched it back to pure yarn. But probably i'm missing something.
All failed containers had the same error:
Container: container_1452447718890_0001_01_000002 on localhost_37976
======================================================================
LogType: stderr
LogLength: 45
Log Contents:
Error: Could not find or load main class 256
Jps results:
7261 SecondaryNameNode
7535 NodeManager
7413 ResourceManager
6928 NameNode
7593 JobHistoryServer
7047 DataNode
7733 QuorumPeerMain
8433 Jps
Main logs:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/yarn/lib/giraph-examples-1.1.0-for-hadoop-2.4.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/mapreduce/giraph-examples-1.1.0-for-hadoop-2.4.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/01/15 21:53:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/15 21:53:50 INFO client.RMProxy: Connecting to ResourceManager at hdnode01/192.168.0.10:8050
16/01/15 21:53:51 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
16/01/15 21:53:51 INFO input.FileInputFormat: Total input paths to process : 1
16/01/15 21:53:52 INFO mapreduce.JobSubmitter: number of splits:1
16/01/15 21:53:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1452905418747_0001
16/01/15 21:53:53 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
16/01/15 21:53:53 INFO impl.YarnClientImpl: Submitted application application_1452905418747_0001
16/01/15 21:53:54 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1452905418747_0001/
16/01/15 21:53:54 INFO mapreduce.Job: Running job: job_1452905418747_0001
16/01/15 21:54:04 INFO mapreduce.Job: Job job_1452905418747_0001 running in uber mode : false
16/01/15 21:54:04 INFO mapreduce.Job: map 0% reduce 0%
16/01/15 21:54:07 INFO mapreduce.Job: Task Id : attempt_1452905418747_0001_m_000000_0, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/01/15 21:54:11 INFO mapreduce.Job: Task Id : attempt_1452905418747_0001_m_000000_1, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/01/15 21:54:15 INFO mapreduce.Job: Task Id : attempt_1452905418747_0001_m_000000_2, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/01/15 21:54:21 INFO mapreduce.Job: map 100% reduce 100%
16/01/15 21:54:21 INFO mapreduce.Job: Job job_1452905418747_0001 failed with state FAILED due to: Task failed task_1452905418747_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
16/01/15 21:54:21 INFO mapreduce.Job: Counters: 12
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=15548
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=7774
Total vcore-seconds taken by all map tasks=7774
Total megabyte-seconds taken by all map tasks=3980288
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
16/01/15 21:54:21 INFO client.RMProxy: Connecting to ResourceManager at hdnode01/192.168.0.10:8050
16/01/15 21:54:22 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
16/01/15 21:54:22 INFO input.FileInputFormat: Total input paths to process : 0
16/01/15 21:54:22 INFO mapreduce.JobSubmitter: number of splits:0
16/01/15 21:54:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1452905418747_0002
16/01/15 21:54:22 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
16/01/15 21:54:22 INFO impl.YarnClientImpl: Submitted application application_1452905418747_0002
16/01/15 21:54:22 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1452905418747_0002/
16/01/15 21:54:22 INFO mapreduce.Job: Running job: job_1452905418747_0002
16/01/15 21:54:32 INFO mapreduce.Job: Job job_1452905418747_0002 running in uber mode : false
16/01/15 21:54:32 INFO mapreduce.Job: map 0% reduce 0%
16/01/15 21:54:36 INFO mapreduce.Job: Task Id : attempt_1452905418747_0002_r_000000_0, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/01/15 21:54:41 INFO mapreduce.Job: Task Id : attempt_1452905418747_0002_r_000000_1, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/01/15 21:54:46 INFO mapreduce.Job: Task Id : attempt_1452905418747_0002_r_000000_2, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/01/15 21:54:51 INFO mapreduce.Job: map 0% reduce 100%
16/01/15 21:54:52 INFO mapreduce.Job: Job job_1452905418747_0002 failed with state FAILED due to: Task failed task_1452905418747_0002_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1
16/01/15 21:54:52 INFO mapreduce.Job: Counters: 10
Job Counters
Failed reduce tasks=4
Launched reduce tasks=4
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=11882
Total time spent by all reduce tasks (ms)=5941
Total vcore-seconds taken by all reduce tasks=5941
Total megabyte-seconds taken by all reduce tasks=3041792
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
I have a problem in mapred-site.xml. My mapred-site.xml was:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdnode01:54311</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>
</property>
<property>
<name>mapreduce.job.maps</name>
<value>4</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>256</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>256</value>
</property>
</configuration>
The two last properties were the problem. Deleting both (or using -Xmx256m instead of 256) solved my problem.
Related
I have a cluster of 1 Master and 1 Slave that are connected and "probably" communicating, I have followed several guides to install and setup the cluster in which almost all of them are similar, only differences are the memory and cores assigned.
Both my master and slave have 8vcores and 32GB each, with around 600GB of SDD
However when I try to run a hadoop task I get the following message:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output
20/11/03 15:51:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/03 15:51:35 INFO client.RMProxy: Connecting to ResourceManager at master/master:8032
20/11/03 15:51:36 INFO input.FileInputFormat: Total input paths to process : 1
20/11/03 15:51:36 INFO mapreduce.JobSubmitter: number of splits:1
20/11/03 15:51:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1604418534431_0001
20/11/03 15:51:36 INFO impl.YarnClientImpl: Submitted application application_1604418534431_0001
20/11/03 15:51:36 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1604418534431_0001/
20/11/03 15:51:36 INFO mapreduce.Job: Running job: job_1604418534431_0001
20/11/03 15:51:43 INFO mapreduce.Job: Job job_1604418534431_0001 running in uber mode : false
20/11/03 15:51:43 INFO mapreduce.Job: map 0% reduce 0%
20/11/03 15:51:46 INFO mapreduce.Job: Task Id : attempt_1604418534431_0001_m_000000_0, Status : FAILED
Exception from container-launch.
Container id: container_1604418534431_0001_01_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
at org.apache.hadoop.util.Shell.run(Shell.java:479)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 1
20/11/03 15:51:49 INFO mapreduce.Job: Task Id : attempt_1604418534431_0001_m_000000_1, Status : FAILED
Exception from container-launch.
Container id: container_1604418534431_0001_01_000003
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
at org.apache.hadoop.util.Shell.run(Shell.java:479)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 1
20/11/03 15:51:52 INFO mapreduce.Job: Task Id : attempt_1604418534431_0001_m_000000_2, Status : FAILED
Exception from container-launch.
Container id: container_1604418534431_0001_01_000004
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
at org.apache.hadoop.util.Shell.run(Shell.java:479)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 1
20/11/03 15:51:57 INFO mapreduce.Job: map 100% reduce 100%
20/11/03 15:51:58 INFO mapreduce.Job: Job job_1604418534431_0001 failed with state FAILED due to: Task failed task_1604418534431_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
20/11/03 15:51:58 INFO mapreduce.Job: Counters: 16
Job Counters
Failed map tasks=4
Killed reduce tasks=1
Launched map tasks=4
Other local map tasks=3
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3946
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=3946
Total time spent by all reduce tasks (ms)=0
Total vcore-milliseconds taken by all map tasks=3946
Total vcore-milliseconds taken by all reduce tasks=0
Total megabyte-milliseconds taken by all map tasks=4845688
Total megabyte-milliseconds taken by all reduce tasks=0
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
What I am trying to do is the following:
echo "hello world hello Hello" > ~/Downloads/test.txt
hadoop fs -mkdir /input
hadoop fs -put ~/Downloads/test.txt /input
hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output
I fixed the problem by changing the hostnames on the hosts file, I had to add an extra one that helped with the resolution of the hostname to the IP of it (apart from master)
Example:
<master.ip> master <hostname>
Thanks everyone for their help!
I'm new to Mahout and Random-Forest.I wanna classify my dataset and have built the random-forest on my three virtual Hadoop nodes.
As we know,first of all,I made the descriptor(/des.info).And I built the classifier(/user/hadoop/forest) .There was an error but it completed successfully.However I was stuck when I tried to test it.
My systems are all CentOS7 with hadoop-3.0.0.
Here is the HDFS:
[hadoop#hadoop1 ~]$ hadoop fs -ls /
Found 5 items
-rw-r--r-- 1 hadoop supergroup 8807688 2018-04-29 19:59 /des.info
-rw-r--r-- 1 hadoop supergroup 79736192 2018-04-29 19:55 /features.txt
-rw-r--r-- 1 hadoop supergroup 278 2018-05-01 03:50 /test.txt
drwx------ - hadoop supergroup 0 2018-04-29 20:05 /tmp
drwxr-xr-x - hadoop supergroup 0 2018-04-29 20:05 /user
Here is the code of the third step:
[hadoop#hadoop1 ~]$hadoop jar /opt/mahout-distribution-0.9/mahout-examples-0.9-job.jar org.apache.mahout.classifier.df.mapreduce.TestForest -i /test.txt -ds /des.info -m /user/hadoop/forest -a -mr -o prediction
2018-05-01 03:53:21,595 INFO mapreduce.Classifier: Adding the dataset to the DistributedCache
2018-05-01 03:53:21,597 INFO mapreduce.Classifier: Adding the decision forest to the DistributedCache
2018-05-01 03:53:21,600 INFO mapreduce.Classifier: Configuring the job...
2018-05-01 03:53:21,605 INFO mapreduce.Classifier: Running the job...
2018-05-01 03:53:21,702 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/192.168.80.100:8032
2018-05-01 03:53:22,073 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1525056498669_0002
2018-05-01 03:53:22,507 INFO input.FileInputFormat: Total input files to process : 1
2018-05-01 03:53:23,009 INFO mapreduce.JobSubmitter: number of splits:1
2018-05-01 03:53:23,170 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-05-01 03:53:23,353 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1525056498669_0002
2018-05-01 03:53:23,355 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-05-01 03:53:23,663 INFO conf.Configuration: resource-types.xml not found
2018-05-01 03:53:23,663 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-05-01 03:53:23,813 INFO impl.YarnClientImpl: Submitted application application_1525056498669_0002
2018-05-01 03:53:23,870 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1525056498669_0002/
2018-05-01 03:53:23,871 INFO mapreduce.Job: Running job: job_1525056498669_0002
2018-05-01 03:53:31,029 INFO mapreduce.Job: Job job_1525056498669_0002 running in uber mode : false
2018-05-01 03:53:31,030 INFO mapreduce.Job: map 0% reduce 0%
2018-05-01 03:53:34,089 INFO mapreduce.Job: Task Id : attempt_1525056498669_0002_m_000000_0, Status : FAILED
Error: java.lang.ArrayIndexOutOfBoundsException: 946827879
at org.apache.mahout.classifier.df.node.Node.read(Node.java:58)
at org.apache.mahout.classifier.df.DecisionForest.readFields(DecisionForest.java:197)
at org.apache.mahout.classifier.df.DecisionForest.read(DecisionForest.java:203)
at org.apache.mahout.classifier.df.DecisionForest.load(DecisionForest.java:225)
at org.apache.mahout.classifier.df.mapreduce.Classifier$CMapper.setup(Classifier.java:209)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:794)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
2018-05-01 03:53:46,822 INFO mapreduce.Job: Task Id : attempt_1525056498669_0002_m_000000_1, Status : FAILED
2018-05-01 03:53:51,342 INFO mapreduce.Job: Task Id : attempt_1525056498669_0002_m_000000_2, Status : FAILED
Error: java.lang.ArrayIndexOutOfBoundsException: 946827879
at org.apache.mahout.classifier.df.node.Node.read(Node.java:58)
at org.apache.mahout.classifier.df.DecisionForest.readFields(DecisionForest.java:197)
at org.apache.mahout.classifier.df.DecisionForest.read(DecisionForest.java:203)
at org.apache.mahout.classifier.df.DecisionForest.load(DecisionForest.java:225)
at org.apache.mahout.classifier.df.mapreduce.Classifier$CMapper.setup(Classifier.java:209)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:794)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
2018-05-01 03:54:05,463 INFO mapreduce.Job: map 100% reduce 0%
2018-05-01 03:54:06,479 INFO mapreduce.Job: Job job_1525056498669_0002 failed with state FAILED due to: Task failed task_1525056498669_0002_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
2018-05-01 03:54:06,556 INFO mapreduce.Job: Counters: 12
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=28056
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=28056
Total vcore-milliseconds taken by all map tasks=28056
Total megabyte-milliseconds taken by all map tasks=28729344
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Exception in thread "main" java.lang.IllegalStateException: Job failed!
at org.apache.mahout.classifier.df.mapreduce.Classifier.run(Classifier.java:127)
at org.apache.mahout.classifier.df.mapreduce.TestForest.mapreduce(TestForest.java:188)
at org.apache.mahout.classifier.df.mapreduce.TestForest.testForest(TestForest.java:174)
at org.apache.mahout.classifier.df.mapreduce.TestForest.run(TestForest.java:146)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.mahout.classifier.df.mapreduce.TestForest.main(TestForest.java:315)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Now I'm confused very much.
Because this is incompatible with hadoop and mahout versions. The Mahout 0.9 random forest algorithm can only run on hadoop 1.x.
I am using Hadoop 2.7.0 in pseudo node mode, on a Fedora 22 Virtual Machine. A few days back the MapReduce jobs ran fine, but after installed Oozie and made modifications to the yarn-site.xml . I am getting the below error on running the Pi example job and come what may I am not able to debug the error,
EDITED - I am running the job using command line and NOT using the oozie workflow engine .. command - hadoop jar 10 100
Starting Job
15/12/17 15:22:05 INFO client.RMProxy: Connecting to ResourceManager at /192.168.122.1:8032
15/12/17 15:22:06 INFO input.FileInputFormat: Total input paths to process : 10
15/12/17 15:22:06 INFO mapreduce.JobSubmitter: number of splits:10
15/12/17 15:22:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1450326099697_0001
15/12/17 15:22:07 INFO impl.YarnClientImpl: Submitted application application_1450326099697_0001
15/12/17 15:22:07 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1450326099697_0001/
15/12/17 15:22:07 INFO mapreduce.Job: Running job: job_1450326099697_0001
15/12/17 15:22:17 INFO mapreduce.Job: Job job_1450326099697_0001 running in uber mode : false
15/12/17 15:22:17 INFO mapreduce.Job: map 0% reduce 0%
15/12/17 15:22:24 INFO mapreduce.Job: map 10% reduce 0%
15/12/17 15:22:30 INFO mapreduce.Job: map 20% reduce 0%
15/12/17 15:22:36 INFO mapreduce.Job: map 30% reduce 0%
15/12/17 15:22:42 INFO mapreduce.Job: map 40% reduce 0%
15/12/17 15:22:46 INFO mapreduce.Job: map 50% reduce 0%
15/12/17 15:22:51 INFO mapreduce.Job: map 60% reduce 0%
15/12/17 15:22:56 INFO mapreduce.Job: map 70% reduce 0%
15/12/17 15:23:01 INFO mapreduce.Job: map 80% reduce 0%
15/12/17 15:23:07 INFO mapreduce.Job: map 90% reduce 0%
15/12/17 15:23:13 INFO mapreduce.Job: map 100% reduce 0%
15/12/17 15:23:18 INFO mapreduce.Job: map 100% reduce 100%
15/12/17 15:23:23 INFO ipc.Client: Retrying connect to server: vlan722-rsvd-router.ddr.priv/192.168.122.1:34460. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
sleepTime=1000 MILLISECONDS)
15/12/17 15:23:24 INFO ipc.Client: Retrying connect to server: vlan722-rsvd-router.ddr.priv/192.168.122.1:34460. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
sleepTime=1000 MILLISECONDS)
15/12/17 15:23:25 INFO ipc.Client: Retrying connect to server: vlan722-rsvd-router.ddr.priv/192.168.122.1:34460. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
sleepTime=1000 MILLISECONDS)
15/12/17 15:23:28 INFO mapreduce.Job: map 0% reduce 0%
15/12/17 15:23:28 INFO mapreduce.Job: Job job_1450326099697_0001 failed with state FAILED due to: Application application_1450326099697_0001 failed 2 times due to AM Container for
appattempt_1450326099697_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://hadoop:8088/cluster/app/application_1450326099697_0001Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1450326099697_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
15/12/17 15:23:28 INFO mapreduce.Job: Counters: 0
Job Finished in 82.924 seconds
Estimated value of Pi is 3.14800000000000000000
yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>
/home/osboxes/hadoop/etc/hadoop,
/home/osboxes/hadoop/share/hadoop/common/*,
/home/osboxes/hadoop/share/hadoop/common/lib/*,
/home/osboxes/hadoop/share/hadoop/hdfs/*,
/home/osboxes/hadoop/share/hadoop/hdfs/lib/*,
/home/osboxes/hadoop/share/hadoop/yarn/*,
/home/osboxes/hadoop/share/hadoop/yarn/lib/*,
/home/osboxes/hadoop/share/hadoop/mapreduce/*,
/home/osboxes/hadoop/share/hadoop/mapreduce/lib/*
</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>5120</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>http://192.168.122.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>http://192.168.122.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>http://192.168.122.1:8031</value>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>http://192.168.122.1:8041</value>
</property>
Any help on this would be very much appreciated.
EDIT - yarn-site.xml before
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
Finally I solved the issue by making the following change to mapred-site.xml ,
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>http://localhost:19888</value>
</property>
After that the jobs ran perfectly fine.
I'm trying to run some hadoop program to extracting keywords of some abstracts in Ubuntu. When I run my program using Hadoop, I get the following error.
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
INFO input.FileInputFormat: Total input paths to process : 1
INFO mapreduce.JobSubmitter: number of splits:1
INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1404812840999_0001
INFO impl.YarnClientImpl: Submitted application application_1404812840999_0001
INFO mapreduce.Job: The url to track the job: http://shiva-VirtualBox:8088/proxy/application_1404812840999_0001/
INFO mapreduce.Job: Running job: job_1404812840999_0001
INFO mapreduce.Job: Job job_1404812840999_0001 running in uber mode : false
INFO mapreduce.Job: map 0% reduce 0%
INFO mapreduce.Job: Job job_1404812840999_0001 failed with state FAILED due to: Application application_1404812840999_0001 failed 2 times due to AM Container for appattempt_1404812840999_0001_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
14/07/08 14:21:44 INFO mapreduce.Job: Counters: 0
What's the cause of this error?
Note that I converted my mapreduce project to maven project for using lucene library in my code.
Is your resource manager really on the /0.0.0.0:8032? It also seams you are not using Toolrunner, so try to rewrite your mapreduce Hadoop: Implementing the Tool interface for MapReduce driver.
Hope it helps
Number of thread increased, JVM memory and CPU is fully utilised. Please increase the JVM size and increase memory limit of Mapper and reducer task.
conf.set("mapreduce.map.memory.mb", "4096");
conf.set("mapreduce.map.java.opts", "-Xmx3500m");
I'm trying to convert some texts to mahout sequence files using:
mahout seqdirectory -i Lastfm-ArtistTags2007 -o seqdirectory
But all I get is a OutOfMemoryError, as here:
Running on hadoop, using /usr/bin/hadoop and HADOOP_CONF_DIR=
MAHOUT-JOB: /opt/mahout/mahout-examples-0.9-job.jar
14/04/07 16:44:34 INFO common.AbstractJob: Command line arguments: {--charset=[UTF-8], --chunkSize=[64], --endPhase=[2147483647], --fileFilterClass=[org.apache.mahout.text.PrefixAdditionFilter], --input=[Lastfm-ArtistTags2007], --keyPrefix=[], --method=[mapreduce], --output=[seqdirectoryjps], --startPhase=[0], --tempDir=[temp]}
14/04/07 16:44:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/07 16:44:35 INFO input.FileInputFormat: Total input paths to process : 4
14/04/07 16:44:35 WARN snappy.LoadSnappy: Snappy native library not loaded
14/04/07 16:44:35 INFO mapred.JobClient: Running job: job_local407267609_0001
14/04/07 16:44:35 INFO mapred.LocalJobRunner: Waiting for map tasks
14/04/07 16:44:35 INFO mapred.LocalJobRunner: Starting task: attempt_local407267609_0001_m_000000_0
14/04/07 16:44:35 INFO util.ProcessTree: setsid exited with exit code 0
14/04/07 16:44:35 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin#6ad3ad65
14/04/07 16:44:35 INFO mapred.MapTask: Processing split: Paths:/home/giuliano/cook/lastfm/Lastfm-ArtistTags2007/README.txt:0+2472,/home/giuliano/cook/lastfm/Lastfm-ArtistTags2007/ArtistTags.dat:0+71652722,/home/giuliano/cook/lastfm/Lastfm-ArtistTags2007/tags.txt:0+1739746,/home/giuliano/cook/lastfm/Lastfm-ArtistTags2007/artists.txt:0+327051
14/04/07 16:44:35 INFO compress.CodecPool: Got brand-new compressor
14/04/07 16:44:35 INFO mapred.LocalJobRunner: Map task executor complete.
14/04/07 16:44:35 WARN mapred.LocalJobRunner: job_local407267609_0001
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:119)
at org.apache.mahout.text.WholeFileRecordReader.nextKeyValue(WholeFileRecordReader.java:118)
at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.nextKeyValue(CombineFileRecordReader.java:69)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
14/04/07 16:44:36 INFO mapred.JobClient: map 0% reduce 0%
14/04/07 16:44:36 INFO mapred.JobClient: Job complete: job_local407267609_0001
14/04/07 16:44:36 INFO mapred.JobClient: Counters: 0
14/04/07 16:44:36 INFO driver.MahoutDriver: Program took 1749 ms (Minutes: 0.02915)
I am using Mahout 0.9, Hadoop 1.2.1 and OpenJDK Java7u25
defining MAHOUT_HEAPSIZE to 4096 did not help, and the text files can be found here: http://static.echonest.com/Lastfm-ArtistTags2007.tar.gz
Currently the spawned job is executed as a local job runner, the execution happens only in the node in which you fired the job. Specify the job tracker address by setting the property mapred.job.tracker in your mapred-site.xml inorder to make the execution distributed.
Execution in distributed mode might solve your outOfMemory issue
If you look at the environment variable HADOOP_CONF_DIR, its values is empty set its value using the following command export HADOOP_CONF_DIR=/etc/hadoop/conf. Make sure the value of the property mapred.job.tracker which should point to your jobTracker in /etc/hadoop/conf/mapred-site.xml configuration