I want to build a cluster environment where nodes are automatically created and deleted. The jobs are to be distributed to the various nodes using Slurm.
Two questions:
Is there an agent or similar for the Slurm workers so that the nodes automatically register with the head node?
Is it possible to change the Slurm config file during runtime? (since new worker nodes could be added or deleted).
You would need to restart the Slurm daemon for changes to the slurm.conf file to take effect, which could be problematic for jobs that are running. You may have errors (job failures or worse) if the Slurm control daemon finds that the slurm.conf is different due to checksum mismatch (see the official docs on adding nodes: https://slurm.schedmd.com/faq.html#add_nodes).
Related
I setup a cluster which consists of 1 master and 3 workers.
In normal condition, as we know, if users submit some jobs and jobs will be distributed to three workers for execution.
However, if I want to assign such as
job id_1 to worker 1 and worker 2, but no worker 3
job id_2 to worker 1, worker 2 and worker 3
job id_3 to worker 2 and worker 3, but no worker 1
Can Spark do this through some configuration setting, scheduling or write code to assign the job to workers which are specified?
Any idea or method can be recommended.
You should not do this because it will make your job slow will create unwanted problems .
Set location preference! If you know all the names of the worker
machines, you can created with a version of parallelize where you can
set the preferred location of each partition. That will ensure a
deterministic behavior of sending each partition to corresponding
worker (assuming speculative execution, and delay scheduling is turned
off).
To figure out the name of the worker nodes without hardcoding, you
could run a dummy Spark job with many many partitions which will
return the hostname of all the workers. Not that will try to ensure
(but not guarantee) at least one partition will be scheduled on each
active worker. In fact, if there are other jobs running in the system,
then probably these dummy tasks will not get scheduled on all the
workers. Hard to get around this without some outside mechanism to
know all the workers in the cluster.
I have never tried this thing the way you are trying to submit the job.
Might be this is a possible solution hint for your questionSpark Reply
Go through the Cluster Mode
Is it possible to execute a set of tuples (based on a particular field) on a particular worker node in the storm topology. Need to minimize network load in the cluster.
You can go for a custom scheduler ... it will allow you to bind a specific task to a supervisor , might worth taking a look into it
I submitted job to the cluster of 4 hosts, I can see that it was correctly spread among 4 nodes, 1 map task per node.
Later on, one of the node failed.
I stopped tasktracker on the failed node, added the ID of that node to excludes file and updated list of nodes with hadoop mradmin -refreshNodes. The failed node disappeared from list of available nodes on hadoop administration pages.
Then I started tasktracker again, updated nodes with mradmin, and observed that the node appeared in job tracker list again.
During the time of the node being down, hadoop re-scheduled map task execution on another node, so it started to run 2 map jobs. I've got the cluster unbalanced:
2 nodes were running 1 task each,
1 node was running 2 tasks
and 1 node (the one I restarted) was running no tasks.
I killed the job with hadoop job -kill-task attempt_201308010141_0001_m_000000_1 and looks like it never starts again - so I can see 3 nodes running 1 task each, 1 node with no tasks at all and 1 pending task in the list.
Am I missing something? What is the correct way of 'moving' task from one node to another one?
Jobs keep a list of blacklisted tasktrackers (there is a global blacklist and a per job one).
I think that's why your new attempt don't start again at the end on the restarted task tracker.
You can try the commands :
hadoop job -unblacklist <jobid> <hostname>
hadoop job -unblacklist-tracker <hostname>
From http://doc.mapr.com/display/MapR/TaskTracker+Blacklisting
I have a Fully-Distributed Hadoop cluster with 4 nodes.When I submit my job to Jobtracker which decide 12 map tasks will be cool for my job,something strange happens.The 12 map tasks always running on a single node instead of running on the entire cluster.Before I ask the question ,I have already done the things below:
Try different Job
Run start-balance.sh to rebalance the cluster
But it does not work,so I hope someone can tell me why and how to fix it.
If all the blocks of input data files are in that node, the scheduler with prioritize the same node
Apparently the source data files is in one data node now. It could't be the balancer's fault. From what I can see, your hdfs must only have one replication or you are not in a Fully-Distributed Hadoop cluster.
Check how your input is being split. You may only have one input split, meaning that only one Node will be used to process the data. You can test this by adding more input files to your stem and placing them on different nodes, then checking which nodes are doing the work.
If that doesn't work, check to make sure that your cluster is configured correctly. Specifically, check that your name node has paths to your other nodes set in its slaves file, and that each slave node has your name node set in its masters file.
I run some batch jobs with data inputs that are constantly changing and I'm having problems provisioning capacity. I am using whirl to do the intial setup but once I start, for example, 5 machines I don't know how to add new machines to it while its running. I don't know in advance how complex or how large the data will be so I was wondering if there was a way to add new machines to a cluster and have it take effect right away(or with some delay but don't want to have to bring down the cluster and bring it up with the new nodes).
There is exact explanation how to add node:
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
In the same time - I am not sure that already running jobs will take advantages of these nodes since planning where to run each task happens during job start time (as far as I understand).
I also think that it is more practical to run Task Trackers only on these transient nodes.
Check the files referred by the below parameters:
dfs.hosts => dfs.include
dfs.hosts.exclude
mapreduce.jobtracker.hosts.filename => mapred.include
mapreduce.jobtracker.hosts.exclude.filename
You can add the list of hosts to the files dfs.include and mapred.include and then run
hadoop mradmin -refreshNodes ;
hadoop dfsadmin -refreshNodes ;
That's all.
BTW, 'mradmin -refreshNodes' facility was added in 0.21
Nikhil