In Microsoft HPC Cluster Manager, is it possible to run two jobs (MPI job) simultaneously on same node? If so, how a job should be configured?
I've made some tries with HPC Cluster Manager, and I've found a solution like that;
First, job scheduler configuration must be selected as Balanced
Second, job resource type must be selected Core or Socket, not Node.
In addition to these two settings, if minimum requested resource is available for both 2 jobs, they start to run simultaneously on same node.
Related
From Hadoop The Definitive Guide
The whole process is illustrated in Figure 7-1. At the highest level,
there are five independent entities:
• The client, which submits the MapReduce job.
• The YARN resource manager, which coordinates the allocation of
compute re‐ sources on the cluster.
• The YARN node managers, which launch and monitor the compute
containers on machines in the cluster.
• The MapReduce application master, which coordinates the tasks
running the Map‐ Reduce job. The application master and the MapReduce
tasks run in containers that are scheduled by the resource manager and
managed by the node managers.
What is the MapReduce application master?
In a MapReduce program written in Java, we need three things: a map function, a reduce function, and some code with main() function to run the job. Is the MapReduce application master the code with main() function to run a map reduce job?
main() function in typical Hadoop program usually does these things:
specifies the input/output path for the job
configures mappers/reducers/combiners/partitioners
configures memory
Then it creates an instance of Job interface, runs it and calls waitForCompletion, which blocks until job is finished. This call sends Yarn application request under the hood, which spawns AppMaster somewhere on the cluster.
AppMaster is responsible for creating Map/Reduce processes, tracking their status and reporting the progress. There's 1 instance of AppMaster for every job running on Hadoop cluster.
Using dc/os we like to schedule tasks close to the data that the task requires that in our case is stored in hadoop/hdfs (on an HDP cluster). Issue is that the hadoop cluster is not run from within dc/os and so we are looking for a way to offer only a subset of the system resources.
For example: say we like to reserve 8GB of memory to data node services, then we like to provide the remainder to dc/os to schedule tasks.
From what i have read so far, the task can specify the resources it requires, but i have not found any means to specify what you want to offer from the node perspective.
I'm aware that a CDH cluster can be run on dc/os, that would be one way to go, but for now that is not provided for HDP.
Thanks for any idea's/tips,
Paul
I know that there is only Resource Manager in a hadoop cluster.
From my understanding, there should be only one Application Master for a cluster as well. Is that right? Following is my understanding of how a mapreduce job is run in YARN. Please correct if my understanding is not right.
Application execution sequence of steps on YARN:
Client submits a job to the Resource Manager (RM). RM runs on Master Node. There is only one RM across the cluster to manage the resources. Resource Manager is a Daemon process.
RM will go to HDFS thru Name Node.
RM spins up an Application Master (AM). AM will reach HDFS thru Name Node. It will create a mapper matrix. This is the mapper phase. Like if Block 1 is available on Name Node 5 or 6.
Based on Mapper matrix information, AM sends requests to individual Node managers (NM) to run a particular task for each block. NM runs on slave node.
Each NM sends a request to RM to get a container. A container executes an application specific process with a constrained set of resources (memory, CPU etc).
Mapper task runs in the container and sends the heart beat to the Application master. AM also sends the heart beat to RM.
After all the processes are done, AM starts another matrix for Reducer tasks.
After all the reducer tasks are completed, the AM sends the results to RM.
RM lets the client know the results and kills the AM.
Application Master can get stuck. That is why it is sending heart beats to Resource Manager
Thanks much
Nath
Other steps look fine.
RM spins up an Application Master (AM). AM will reach HDFS thru Name Node. It will create a mapper matrix. This is the mapper phase. Like if Block 1 is available on Name Node 5 or 6.
Slight correction here. The AM can only execute inside any given container. So first the RM requests a node manager on some node to start a container and then only the AM gets launched inside that cotainer, not before. So there will be a container dedicated to the AM.
Is it allowed in YARN to have "multiple containers" of the "same application" running on one DataNode?
Yes.
Example: multiple mappers of a job running on same DN
Yes, any data node can have multiple containers running in parallel.
The number of parallel containers is calculated by the YARN resource manager by considering the amount of ram memory , cpu cores available on the data node.
There are chances to see multiple containers running on the same data node when resource manager decides to run multiple mappers/reducers on the containers of a data node.
I want to find out IPs of slave nodes where currently map reduce job is running or about to run for a given Job.
Is there any method to do this ?
Thanks in Advance.
For any job, you can view the list of running tasks through the Job Scheduler Web UI - this will detail the nodes on which the task is running.
As for where tasks are about to run - this is not neccessarily decided in advance. As slots become available on a node, the Job Scheduler (there are a number which behave differently depending on your needs) identifies a job task which will run on that node (based upon a number of criteria, hopefully honoring data locality where it can) and instructs the task tracker on that node to run the specific task.
Programatically, look at the javadocs for the JobClient class, it should be able to acquire information about the running tasks, and their node names (you'll probably need to do a DNS lookup to get the actual IPs i imagine)
Hadoop comes with several web interfaces which are by default (see conf/hadoop-default.xml) available at these locations:
http://localhost:50030/ – web UI for MapReduce job tracker(s)
http://localhost:50060/ – web UI for task tracker(s)
http://localhost:50070/ – web UI for HDFS name node(s)
Thanks to #Chris..
Programatically, look at the javadocs for the JobClient class, it should be able to acquire information about the running tasks, and their node names.