Job History server is going out of memory when trying to load the task status after the job completed. We are trying to increase the memory for the job history server. Any idea how we can increase the XMX for the job history server? I am Apache Hadoop 2.4.0 and running YARN.
Restart the history server with the following settings:
HADOOP_JOB_HISTORYSERVER_HEAPSIZE=5 bash -x bin/mapred --config /Users/ajaymysore/gitHome/proton/resources/hadoop/conf historyserver
Note that 5 is the amount of memory you want to allocate in MB.
Related
After applications finished on yarn, a great amount application history will be shown in yarn.
Both in web-ui http://localhost:8088/cluster and command line yarn application -list -appStates FINISHED
How can I clean these history? Or where does yarn store these history information?
Restart RM,NM does not clean these history.
Delete the logs in local file system ${YARN_LOG_DIR}/userlogs does not clean these history.
Delete the logs' directory/files in hdfs:///tmp/logs/username/logs does not clean these history.
Hadoop version: 2.7.7
If you've enabled log-aggregation, you can set yarn.log-aggregation.retain-seconds to a reasonable value (like a day or a week depending on how many jobs you run) to have YARN purge jobs on a continual basis.
Otherwise set yarn.nodemanager.log.retain-seconds to determine how long you want to keep logs.
You can also disable persistence on restarts by setting yarn.resourcemanager.recovery.enabled to false in your yarn-site.xml.
Refer to yarn-default.xml for all the log configurables in YARN and ResourceManager Restart for more information on persistence.
Oh, I've found the solution.
yarn resourcemanager -format-state-store
and restart the cluster.
But still don't know where yarn store these history, in ZK?
I have a hadoop cluster 2.7.4 version. Due to some reason, I have to restart my cluster. I need job IDs of those jobs that were executed on cluster before cluster reboot. Command mapred -list provide currently running of waiting jobs details only
You can see a list of all jobs on the Yarn Resource Manager Web UI.
In your browser go to http://ResourceManagerIPAdress:8088/
This is how the history looks on the Yarn cluster I am currently testing on (and I restarted the services several times):
See more info here
I am getting "Container... is running beyond virtual memory limits" error while running spark job in yarn cluster mode.
It is not possible to ignore this error or increase Vmem Pmem ratio.
Job is submitted through spark-submit with " --conf spark.driver.memory=2800m".
I think it is because default value of yarn.app.mapreduce.am.command-opts is 1G, so yarn kills my driver/AM as soon as my driver/AM uses more than 1G memory.
So I would like to pass "yarn.app.mapreduce.am.command-opts" to spark-submit in bash script. Passing it with "spark.driver.extraJavaOptions" errors out with "Not allowed to specify max heap(Xmx) memory settings through java options"
So how do I pass it ?
EDIT: I cannot edit conf files as that will make the change for all MR and spark jobs.
I have been trying to run a pig job with multiple steps on Amazon EMR. Here are the details of my environment:
Number of nodes: 20
AMI Version: 3.1.0
Hadoop Distribution: 2.4.0
The pig script has multiple steps and it spawns a long-running map reduce job that has both a map phase and reduce phase. After running for sometime (sometimes an hour, sometimes three or four), the job is killed. The information on the resource manager for the job is:
Kill job received from hadoop (auth:SIMPLE) at
Job received Kill while in RUNNING state.
Obviously, I did not kill it :)
My question is: how do I go about trying to identify what exactly happened? How do I diagnose the issue? Which log files to look at (what to grep for)? Any help on even where the appropriate log files would be greatly helpful. I am new to YARN/Hadoop 2.0
There can be number of reasons. Enable debugging on your cluster and see in the stderr logs for more information.
aws emr create-cluster --name "Test cluster" --ami-version 3.9 --log-uri s3://mybucket/logs/ \
--enable-debugging --applications Name=Hue Name=Hive Name=Pig
More details here:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-debugging.html
I want to ask about hadoop YARN. For example before I start the daemon I have free memory "X" MB, so after I do start-all.sh to start the daemon, my free memory become "Y" MB. So I want to know in particular what service or something that consume my node memory?
Thanks all....
start-all.sh script starts the HDFS daemons: NameNode and DataNode, and YARN daemons: ResourceManager and Nodemanager. Enter the "jps" command, this will list all the running java processes. These daemons consume your CPU Memory.