Hi
hi i want to implement hadoop in cloudsim by netbeans software.
could you please guide me.
meanwhile i already have source of hadoop classes.
CloudSim is a simulator and it does not perform any actual task. Hadoop is a framework which is used to solve large problems using master slave approach. If you want to deploy Hadoop framework on cloud then try making a private cloud using openstack, then create instances in it and then try to deploy Hadoop in it. It is not possible to deploy hadoop framework in cloudsim as it's just a simulator
Related
I´m not sure that someone can help me but I´ll take a try.
I´m running Jenkins on an Openshift-Cluster to use it for Deployment and as a jobserver for running ETL-Jobs. These jobs are transferring data from flatfiles to databases and from db to db.
Now, I should expand the system to transfer data to a hadoop cluster using MapR.
What I would like to know is, how can I use a new Jenkins-Slave as a jobserver on an EdgeNode from the hadoop-cluster using MapR. Do I need the Jenkins on the EdgeNode or am I able to use MapR from my existing Jenkins-Jobserver?
Mabye, someone is able to help me or has some informations/links how to solve it.
Thx to all....
"Use MapR" isn't quite clear to me because I just view it as Hadoop at the end of the day, but you can effectively make your Jenkins slave an "edge node" by installing only the Hadoop Java (maybe also MapR) client utilities plus any XML configuration files from the other edge nodes that define how to communicate with the cluster.
Then, Jenkins would be able to run sh("hadoop jar app.jar"), for example
If you're using Openshift, you might also try putting a Hadoop client inside a Docker image that could run in Jenkins, or anywhere else
I have been using a Hadoop cluster, created using Google's script, for a few months.
Every time I boot the machines I have to manually start Hadoop using:
sudo su hadoop
cd /home/hadoop/hadoop-install/sbin
./start-all.sh
Besides scripting, how can I resolve this?
Or is this just the way it is by default?
(The first boot after cluster creation always starts Hadoop automatically, why not always?)
You have to configure using init.d.
Document provide more details and sample script for datameer. You need to follow similar steps. Script should be smart enough to check all the nodes in the cluster are up before invoking this script using ssh.
While different third-party scripts and "getting started" solutions like Cloud Launcher have varying degrees of support for automatic restart of Hadoop on boot, the officially supported tools are bdutil as a do-it-yourself deployment tool, and Google Cloud Dataproc as a managed service, both of which are already configured with init.d and/or systemd to automatically start Hadoop on boot.
More detailed instructions on using bdutil here.
I have seen HortonWorks put the full Hadoop inside a docker that allows to install Hadoop in different environments. But how about the individual apps inside Hadoop that run on YARN? Especially in a multi-tenant environment, this would be useful.
Appreciate any thoughts on how to achieve this.
As a part of my college project, I would like to modify Hadoop's source code. However, the problem is that I would need atleast 20 systems to test it. Is it possible to setup this modified version of Hadoop in public clouds such as Google Cloud platform or Amazon Services?Can you give me an idea on the procedure to follow?I could only find information about setting up the original Hadoop versions in the public cloud set up. I couldn't find any information that is relevant to my case.Please do help me out.
Amazon offers elastic mapreduce. But as you correctly pointed out you will not be able to deploy your version of hadoop there.
But you still can use Amazon or Google cloud to just get the base linux servers and install your hadoop on it. It is just a longer process but not different from any other hadoop installation if you have done it before.
i am new to Hadoop ,i likes to go in hadoop administration line so studied basics of hadoop and tried to install hadoop in pseudo distribution mode and installed successfully and run some basic examples also, now i need to improve me further,so i need to try a way to learn hadoop installation and configuration in real time so decided to go for Amazon micro instance ,can any one please tell how to install and configure hadoop in Amazon cloud.
Thanks in Advance.
I have tried this personally and you will not really be able to use hadoop on a single micro instance due to memory restrictions. IMHO you should atleast try a medium instance to run hadoop or better yet use their elastic-mapreduce api which is a modified version of hadoop. You can run a 3 node cluster for around 00.25 cents an hour. If you really want to learn big data this is the way I went.
You should check out their documentation here
http://aws.amazon.com/documentation/elasticmapreduce/