I am learning Mapreduce and Hadoop now. I know I can do some tests and run some samples on a singe node. But I really want to do some practice on a real distributed environment. So I want to ask :
Is there a website which can offer a distributed environment for me to do some experiments?
Somebody told me that I can use Amazon web service to build a distributed environment. Is it real? Does someone have such an experience?
And I want to know how you guys learn hadoop before you use it in your work?
Thank you!
There are a few options:
If you just want to learn about the Map/Reduce paradigm, I would recommend you take a look at JSMapReduce. This is embedded directly in the browser, you have nothing to install, and you can create real Map/Reduce programs.
If you want to learn about Hadoop specifically, Amazon has this thing called Elastic Map Reduce which is essentially Hadoop running on AWS, so this enables you to write your Hadoop job, decide how many machines you want in your cluster, which type of machines you want, and then run it, and EMR will do everything, bootstrap the machines for you, run your job and store the results on S3. I would recommend looking at this tutorial to get an idea how to setup a job on EMR. Just remember, EMR is not free, so you'll have to pay for your computing resources.
Alternatively if you're not looking to pay the cost of EMR, you could always setup Hadoop on your local machine in non-distributed mode, and experiment with it, as described here. Even if it's a single node setup, the abstractions will be the same as if you were using a big cluster, so it's a good way to get up to speed and then go on EMR or a real cluster when you want to get serious.
Amazon offers a free tier, so you can spin up some vms and try experimenting that way. The micro instances they have aren't very powerful, but are fine for small scale tests.
You can also spin up VMs on your desktop if it is powerful enough. I have done this myself using VMPlayer. You can install any flavor of Linux you like for free. Ubuntu is pretty easy to start with. When you setup the networking for your VMs, be sure to use bridged networking. That way each VM will get its own IP address on your network so they can communicate with each other.
Well, it's maybe not about '100% online' but should give really good alternative with some details.
If you are not ready to pay for online cluster resources (such as EMR solution mentioned here) and you don't like to build your own cluster but you are not satisfied with single node setup, you can try to build virtual cluster on powerful enough desktop.
You need minimun 3 VM, I prefer Ubuntu. 4 is better. To see real Hadoop you need minimal replication factor 3. So you need 3 dataNode, 3 taskTrackers. Well, you also need nameNode / JobTracker - it could be one of nodes used for dataNode but I'd recommend to have separate VM. If you need HBase, for example, you again need one Master and minimum 3 RegionServer. So, again, you need 3 but better 4 VM,
There is pretty good free product, Cloudera CDH which is 'somewhat commercial' Hadoop distribution. They also have manager with GUI and simplified installation. BTW they have even prepared demo VMs but I never have used them. You can download everything here. They also host lot of materials about Hadoop and their environment.
Alternative between completely free solution with VMs on desktop and paid service like EMR is your virtual cluster built on top of one dedicated server if you have spare. This is what I personally did. One physical server powered by VmWare free solution, 4 virtual machine, 1 SSD for OS and 3 'general' HDD for storages. Every VM runs Ubuntu 11.04 (again free). Cloudera manager free edition, CDH. So everything is free but you need some hardware that is often available as spare. And you have playground. OK, you need to invest time but by my mind you will get greatest experience from this approach.
Although I do not know much about it, another option may be Greenplum's analytic workbench (1000 node cluster w/ Hadoop for testing): http://www.greenplum.com/solutions/analytics-workbench
Related
I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.
Each of them has a laptop, with a quite good amount of memory and processors.
I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.
So far, what I think about is to:
create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node
ask each of them to run the VM on their computer
creating a cluster on top of this network of VMs
However, I have some locks:
1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?
2/ What could be wrong with this approach?
3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?
By the way, I am pretty ok with Hadoop installation and so on...
Thanks in advance for your help, suggestions, ...
is it possible to create a private network of VMs located on their different machines
Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables
so that the hadoop cluster is isolated from the network that links the physical machines?
Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.
What could be wrong with this approach?
You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"
Is there a better way for handling this need of setting a Hadoop cluster
Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5
we are looking to build a cluster of Compute Nodes for Deep Learning model training jobs, some of them on the cloud and others locally, that have NVIDIA GPUs in them. We felt that using Mesos and the framework Marathon (M&M) would be our best options to schedule the cluster. However the documentations for (M&M) seem to be very ambiguous (or at least to me, sorry I'm an intern) and I'm running into a lot of issues concerning Zookeeper and the connections between the nodes.
Plus, it seems like Mesosphere are giving much more importance to DC/OS when it comes to tutorials and docs, and I guess it will also be patched more regularly and its interfaces (GUI and CLI) look much more user-friendly.
So I was wondering if by dropping the exploration of (M&M) and moving to DC/OS, would we lose a lot of control over the cluster? In M&M do we have perks that cannot be given in the Open Source Edition of DC/OS? like monitoring the machines, logging results etc.. If I ask my manager we might also get the Enterprise edition so that's not really a problem, but does DC/OS apply an abstraction layer that isn't really preferable to advanced users?
DC/OS is build around Apache Mesos and Marathon and gives a good default setup for zookeeper, networking, .... So IMO it is a good place to start as you can still use all M&M and Mesos features + the DC/OS features and ease of setup.
Disclaimer: I am working for Mesosphere.
I have to use Hadoop for my research work and I am deciding for the best option to start with. So far I have end up to go with Cloudera. I've downloaded the quick start VM
and started learning different turorials.
The issue is that my system can't afford to run it and perform very slow and I think it might just stop working after I feed it with all the data and run other services.
I was advised to go for a cloud service with 4 cluster node. Can someone please help me by providing the best option and estimated pricing to consider? 1 year plan might be enough to complete my research.
Thanks.
If you are a linux user, Just download the individual components(like hdfs, MR1, YARN, Hbase, Hive etc...) from this Cloudera Archives instead of loading Cloudera Quickstart VM.
If you want to try the 4 node cluster, easiest option is to use cloud.
There are plenty of cloud providers. I have personally used AWS, Google Cloud, Microsoft Azure, IBM SmartCloud. Out of which, AWS is the best to start with.
It is like pay as you go(use).I can recommend you to use a decent EC2 Machine(4 X m3.large Machines)
Type: m3.large CPU:2 RAM:7.5G Storage: 1 x 32 SSD Price: $0.133 per Hour AWS Pricing
If you plan to do the research for one year, I recommend you to go for VPC.
Cons of AWS EC2:
If you launch a machine in EC2, the moment you restart your machine, Your IP and the hostname will get changed.
In AWS VPC, your IP and hostname will remain the same.
If you use 4 Machinesx24x7xone month,it costs you $389.44.
You can calculate the AWS cost by yourself
As best as I can see you have two paths:
Setup Hadoop in a cloud service provider (i.e. Amazon's EC2 or
Redhat's Openshift.
Use Hadoop-as-a-service (i.e. Amazon's EMR or Microsoft's HDInsight).
The first path, setting up Hadoop in a cloud service provider will require you to become a semi-competent Hadoop administrator. If that's your goal, great! However you'll spend a great deal of time learning the necessary skills and mindset to become that. I don't suspect that that is your goal.
The second path is the one I'd recommend out of these two. Using Hadoop-as-a-service you will get up and running faster, but will cost more up front and on an ongoing (per hour basis). You'll still probably save money because you'll be spending less time troubleshooting your Hadoop cluster and more time doing the work you wanted to do in the first place.
I have to wonder, if you can even fit your dataset on your laptop, why are you using big data tools in the first place? True, they'll work. However Big Data is at least partially defined as data sets and computational problems that just don't fit on a single machine.
I am researching now in the topic of improving the MapReduce scheduler but unfortunately my university does not provide a cluster for research purposes. I was thinking about renting a cluster and I heard about Amazon EC2, but I have no experience with its services and I do not know how to use them.
I am in need of 5 machines with the following specifications (for each machine):
A dual-processor (2.2 GHz AMD Opteron(m) Processor 4122 with 4 physical cores)
8GB of RAM
500GB disk
I want to setup the Linux operating system and the Hadoop framework manually, just like I would if I had the machines physically on my hands. I would like to know if Amazon EC2 offers something like this, and I would like to estimate the cost of this infrastructure for, let's say, a month.
In the case I choose Amazon's Elastic MapReduce framework, would I be able to control de version of Hadoop? Could I also be able to change the configuration of the scheduler in it so that I can set my algorithm?
Finally, I would like to know if there is any kind of simulator for MapReduce to make different experiments.
Please excuse my multiple questions, I am new in this field and any guidance would be really appreciated.
I was thinking about renting a cluster and I heard about Amazon EC2, but I have no experience with its services and I do not know how to use them.
Amazon's AWS has a elaborate documentation, for reference here is the Getting Started link to get you going. Also, AWS self-paced labs are worth checking out.
I am in need of 5 machines with the following specifications (for each machine): A dual-processor, 8GB of RAM, and 500GB of disk.
Amazon's AWS provides a wide range of EC2 instance types. Choose which one best fits your use-case from a list of instance types.
I want to setup the Linux operating system and the Hadoop framework manually, just like I would if I had the machines physically on my hands. I would like to know if Amazon EC2 offers something like this, and I would like to estimate the cost of this infrastructure for, let's say, a month.
AWS does not provide a VM without an OS installed in it. All the VM's provided by AWS are pre-loaded with OS and you could manually install Hadoop on top of that. Of course AWS provides a wide range of OS.
Amazon AWS also provides a Simple Monthly Calculator to calculate how much your cluster might cost based on the instances you have selected and number of EB2 volumes you have attached to each instance.
In the case I choose Amazon's Elastic MapReduce framework, would I be able to control de version of Hadoop? Could I also be able to change the configuration of the scheduler in it so that I can set my algorithm?
If you are using AWS EMR to deploy Hadoop cluster then you could select the version of Hadoop to be installed, supported Hadoop versions by Amazon are 2.4.0, 2.2.0, 1.0.3, 0.20.205.
Finally, I would like to know if there is any kind of simulator for MapReduce to make different experiments.
I did not understand about the mapreduce simulator part though.
We want to use an ActiveMQ master/slave configuration based on a shared file system on Amazon EC2 - that's the final goal. Operating system should be Ubuntu 12.04, but that shouldn't make too much difference.
Why not a master/slave configuration based on RDS? We've tried that and it's easy to set up (including multi-AZ). However, it is relatively slow and the failover takes approximately three minutes - so we want to find something else.
Which shared file system should we use? We did some research and came to the following conclusion (which might be wrong, so please correct me):
GlusterFS is often suggested and should be supporting multi-AZs fine.
NFSv4 should be working (while NFSv3 is said to corrupt the file system), but I didn't see too many references to it on EC2 (rather: asked for NFS, got the suggestion to use GlusterFS). Is there any particular reason for that?
Ubuntu's Ceph isn't stable yet.
Hadoop Distributed File System (HDFS) sounds like overkill to me and the NameNode would again be a single point of failure.
So GlusterFS it is? We found hardly any success stories. Instead rather dissuasive entries in the bug tracker without any real explanation: "I would not recommend using GlusterFS with master/slave shared file system with more than probably 15 enqueued/dequeued messages per second." Does anyone know why or is anyone successfully using ActiveMQ on GlusterFS with a lot of messages?
EBS or ephemeral storage? Since GlusterFS should replicate all data, we could use the ephemeral storage or are there any advantages of using EBS (IMHO snapshots are not relevant for our scenario).
We'll probably try out GlusterFS, but according to Murphy's Law we'll run into problems at the worst possible moment. So we'd rather try to avoid that by (hopefully) getting a few more opinions on this. Thanks in advance!
PS: Why didn't I post this on ServerFault? It would be a better fit, but on SO there are 10 times more posts about this topic, so I stuck with the flock.
Just an idea.... but with activemq 5.7 (or maybe already 5.6) you can have pluggable lockers (http://activemq.apache.org/pluggable-storage-lockers.html). So it might be an option to use the filesystem as storage and RDS as just a locking mechanism. Note I have never tried this before.