I have small jar file which is correct because i tested it on other computer and it works with hadoop.
Now i have setup hadoop on my pc and when i submit a job then it never passes the accept state.
In the browser i can see that the job is accepted but it never gets executed. Here is the screenshot.
I see there is a warning in the console:
WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
Full logs are :
C:\Users\afraz\Desktop\MapReduceData>hadoop jar outs.jar 1902 spo
2019-05-01 22:27:40,842 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2019-05-01 22:27:41,882 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-05-01 22:27:41,925 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/afraz/.staging/job_1556742397967_0001
2019-05-01 22:27:42,890 INFO input.FileInputFormat: Total input files to process : 1
2019-05-01 22:27:43,048 INFO mapreduce.JobSubmitter: number of splits:1
2019-05-01 22:27:43,250 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1556742397967_0001
2019-05-01 22:27:43,254 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-05-01 22:27:43,543 INFO conf.Configuration: resource-types.xml not found
2019-05-01 22:27:43,544 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-05-01 22:27:44,093 INFO impl.YarnClientImpl: Submitted application application_1556742397967_0001
2019-05-01 22:27:44,155 INFO mapreduce.Job: The url to track the job: http://LAPTOP-PN52M98R:8088/proxy/application_1556742397967_0001/
2019-05-01 22:27:44,157 INFO mapreduce.Job: Running job: job_1556742397967_0001
Any help would be great.
Seems you have no active NodeManagers
Memory Total: 0B VCores Total: 0
Unhealthy Nodes: 1
Your job was accepted by the ResourceManager, but cannot be ran until there are available resources to start it on
I suggest finding the NodeManager log file on your machine, then seeing if there are any noticable exceptions mentioned there
Related
I was running hadoop program (wordcount) in Horton sandbox. And the situation occurred as below. Especially, this is the program I had ran successfully for many times on exactly the same virtual machine I used, however this time it "failed" without any notification, so it just stuck there. I tried other mapreduce program, the results are similar. Normally, the command lines will notify me with ubermode : false, follows by the Running job..., but this time, it doesn't, and out of no reason.
[root#sandbox ~]# hadoop jar testWC.jar testWC.WCdriver /data/input/pg103.txt /data/output/WC
WARNING: Use "yarn jar" to launch YARN applications.
16/03/11 19:20:01 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
16/03/11 19:20:01 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/10.0.2.15:8050
16/03/11 19:20:01 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/03/11 19:20:02 INFO input.FileInputFormat: Total input paths to process : 1
16/03/11 19:20:02 INFO mapreduce.JobSubmitter: number of splits:1
16/03/11 19:20:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1457723341319_0002
16/03/11 19:20:03 INFO impl.YarnClientImpl: Submitted application application_1457723341319_0002
16/03/11 19:20:03 INFO mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1457723341319_0002/
16/03/11 19:20:03 INFO mapreduce.Job: Running job: job_1457723341319_0002
The program just could not move on anymore.
I tried to run the example program present in Hadoop. However, I'm not successful in getting the output.
I have included my logs below. Please help in solving the issue.
hdfs#localhost:~$ hadoop jar '/opt/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar' wordcount /README.txt /ooo
15/08/21 09:48:26 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8050
15/08/21 09:48:28 INFO input.FileInputFormat: Total input paths to process : 1
15/08/21 09:48:28 INFO mapreduce.JobSubmitter: number of splits:1
15/08/21 09:48:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1440130528838_0001
15/08/21 09:48:29 INFO impl.YarnClientImpl: Submitted application application_1440130528838_0001
15/08/21 09:48:29 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1440130528838_0001/
15/08/21 09:48:29 INFO mapreduce.Job: Running job: job_1440130528838_0001
The mapreduce seems working, there is no error logs which appears.
1/ Can you please detail furthermore your logs?!
2/ Your output folder /ooo is created?? If yes what its contents?!
3/ Verify please if your input file is not empty.
I have a mapreduce job and I ran it with YARN mode. But why my mapreduce job stop and not continue while running job step? It's like this :
15/04/04 17:18:21 INFO impl.YarnClientImpl: Submitted application application_1428142358448_0002
15/04/04 17:18:21 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1428142358448_0002/
15/04/04 17:18:21 INFO mapreduce.Job: Running job: job_1428142358448_0002
And that's stop here. Is because lack of memory? After start-all.sh and all daemon have started, I have about 300-350 MB memory. I need your suggest all, why this happened?
Thanks all..
No, this isn't because of out of memory, else the logs would have clearly mentioned that. The job seems to be in running state and has got stuckup somewhere, you can probably go and check on the application master for more details about the job.
I'm sorry but you mean this thing ?
15/04/05 14:11:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/05 14:11:29 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.1.163:8050
15/04/05 14:11:30 INFO input.FileInputFormat: Total input paths to process : 1
15/04/05 14:11:31 INFO mapreduce.JobSubmitter: number of splits:1
15/04/05 14:11:31 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1428216622742_0003
15/04/05 14:11:31 INFO impl.YarnClientImpl: Submitted application application_1428216622742_0003
15/04/05 14:11:31 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1428216622742_0003/
15/04/05 14:11:31 INFO mapreduce.Job: Running job: job_1428216622742_0003
or something else? on my master node port 8088 there are only tables....
i have a mapreduce job which i submitted with mapreduce.execution.framework=yarn with the following command .
yarn jar dataquality.jar ey_id.dataquality /user/prasanna/ipfile /user/prasanna/opfile
the job is submitted with a job_id and shows that it is running , but it stays still..
[root#cch1utbig3 prasanna]# yarn jar dataquality.jar ey_id.dataquality /user/prasanna/ipfile /user/prasanna/opfile
15/01/29 12:45:16 INFO client.RMProxy: Connecting to ResourceManager at localhost/172.25.121.245:8050
15/01/29 12:45:17 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/01/29 12:45:18 INFO input.FileInputFormat: Total input paths to process : 1
15/01/29 12:45:18 INFO mapreduce.JobSubmitter: number of splits:1
15/01/29 12:45:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1422468943255_0006
15/01/29 12:45:19 INFO impl.YarnClientImpl: Submitted application application_1422468943255_0006
15/01/29 12:45:19 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1422468943255_0006/
15/01/29 12:45:19 INFO mapreduce.Job: Running job: job_1422468943255_0006
.. am i missing anything here ?
Mapreduce with YARN fail to move ahead of 0% map and 0% reduce. I am using Cloudera CDH on google compute high memory instance(13 GM RAM). 8 GB free ram is available on the machine. Can you please help me to fix it?
sunny#hadoop-m:~$ hadoop jar /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/hadoop-mapreduce-examples-2.5.0-cdh5.3.0.jar grep input output 'dfs[a-z.]+'
14/12/24 00:13:53 INFO client.RMProxy: Connecting to ResourceManager at hadoop-m.c.sunny-hadoop-trial.internal/10.240.253.233:8032
14/12/24 00:13:53 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
14/12/24 00:13:54 INFO input.FileInputFormat: Total input paths to process : 5
14/12/24 00:13:54 INFO mapreduce.JobSubmitter: number of splits:5
14/12/24 00:13:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1419360146634_0001
14/12/24 00:13:54 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
14/12/24 00:13:54 INFO impl.YarnClientImpl: Submitted application application_1419360146634_0001
14/12/24 00:13:55 INFO mapreduce.Job: The url to track the job: http://hadoop-m.c.sunny-hadoop-trial.internal:8088/proxy/application_1419360146634_0001/
14/12/24 00:13:55 INFO mapreduce.Job: Running job: job_1419360146634_0001
Resource Manager Output
Some more info about job
yarn-site.xml: http://pastebin.mozilla.org/8113782
mapred-site.xml: http://pastebin.mozilla.org/8113813
Server 's IP got changed because of DHCP service. Client configuration for HDFS and YARN became stale. I needed to update client configuration, I did it with Cloudera manager and now cluster is running fine.