I am trying to form memsql cluster with 3 ec2 instances using memsql ops By adding hosts. Its showing me the following error "The primary MemSQL Ops agent is using a non-standard install directory for MemSQL. If desired, edit each agent to modify their install directory." I have installed memsql using docker on mac as shown in the guide.
I am sorry if I have done something stupid, i am completely alien to this and trying to learn
Thanks
I know you solved the problem, but in case any else has questions, our documentation for EC2 is here: http://docs.memsql.com/latest/setup/setup_cloud/.
maybe this is the same question going on in our chat line.
in general we recommend using cloud.memsql.com for EC2
Related
I am beginning to install hadoop on a cluster. I have ssh access to these machines and I have already installed fabric on them. I was wondering if someone has already written a fabfile to install and deploy hadoop to a cluster easily.
I found this project [0]; but this is written for deploying over AWS instances. I was looking for something where I can just fill in the IPs of my machines and then execute a set of fab commands to bring up the cluster.
[0] http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide/#ec2-deployment-with-fabric-script
I'm AlexJF, the author of the scripts you linked.
The scripts you reference can also be used outside EC2. You just need to configure, as you requested, the list of hosts and configurations on the top of the fabfile.py. Be sure to set EC2 = False (which just happens to be the default).
You'll then have several useful commands available to you.
I successfully deployed Kubernetes on AWS using "getting started on AWS ec2 guide" (http://kubernetes.io/v1.0/docs/getting-started-guides/aws.html), but the disk size of all the minions (kubernetes hosts) is 8gb. I would like to increase the disk size, but I haven't found a way to do it.
I can change the VM size by setting MINION_SIZE (e.g. export MINION_SIZE=m3.medium) prior to installing, but the disk size is still 8gb.
From the Kubernetes install instructions for other cloud providers there's an option to set MINION_DISK_SIZE to set the disk size. I tried that with AWS ec2 installation, and the variable is ignored.
I also poked around the config files, but I didn't see anything obvious.
Any suggestions on how to set the disk size for minions when installing Kubernetes on AWS ec2?
I recently stumbled upon the same issue. Have a look at BLOCK_DEVICE_MAPPINGS in kubernetes/cluster/aws/util.sh. You can modify it to have something more appropriate for a EBS-only minion.
For example:
[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":80}}]
AWS docs: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html
I have faced this very issue and tried the current accepted answer but it looks like Kubernetes is changing quite fast what may make this answer also outdated soon.
To this date, I've tested the solution below that might become or not a definitive solution in the future:
There is this PR on Kubernetes' github project that implements an easy way to ignore the SSD storage by setting KUBE_AWS_STORAGE=ebs before running kubernetes/cluster/kube-up.sh.
Hope it is helpful!
I have posted a similar question earlier but answers to that
Question didn't help me .
I want to setup a three node Hadoop cluster on AWS using cloudera . I have refereed several links where there is step by step guide how to step these clusters :
Link1
Link2
I all these links the step mentioned i have tried but since these installtion are done Instance Stored based Instances. I dont have an option to stop these instances one installation is completed only option to (rebbot/terminate) . I have also tried launching three EBS based instance first and then started installation one of them then tried to add the other two instances while installation like this . But it doesn't detect the other two instances as this data node and I am not sure what other approach to use . Any answers in this regard is would be really helpful .
Specify Private IPs for the hosts to install Cloudera Agent on.
Hi i have a small doubt , I have started to use in my curiosity but now i have the following problem
My scenario is like this - i have 10 machines connected in LAN and i need to create Name Node in one system and Data Nodes in remaining 9 machines . So do i need to install Hadoop on all the 10 machines ?
For example i have ( 1.. 10 ) machines , where machine1 is Server and from machine(2..9) are slaves[Data Nodes] so do i need to install hadoop on all 10 machines ?
And i have searched a lot On Hadoop cluster network on commodity machine but i dint get any thing related to Installation [ that is configuration]. Some of them given like how to config and install Hadoop on own system but not on the clustered environment
Can any one help me ? and give me the detailed idea or article suggested links to do the above process
Thanks
Yes, you need Hadoop installed in every node and each node should have the services started as for appropriate for its role. Also the configuration files, present on each node, have to coherently describe the topology of the cluster, including location/name/port for various common used resources (eg. namenode). Doing this manually, from scratch, is error prone, specially if you never did this before and you don't know exactly what you're trying to do. Also would be good to decide on a specific distribution of Hadoop (HortonWorks, Cloudera, HDInsight, Intel, etc)
I would recommend use one of the many deployment solutions out there. My favorite is Puppet, but I'm sure Chef will do too.
A different (perhaps better?) alternative is to use Ambari, which is a Hadoop specialized deployment and administering solution. See Deploying and Managing Hadoop Clusters with AMBARI.
Some Puppet resources to get you started: Using Vagrant, Puppet, Testing & Hadoop
Please verify below tutorial
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
Hope it helps
Yes hadoop needs to be there on all the computers
For clustered Environment please go through the video
I would like to test out Hadoop & HBase in Amazon EC2, but I am not sure how complicate it is. Is there a stable community AMI that has Hadoop & HBase installed? I am thinking of something like bioconductor AMI
Thank you.
I highly recommend using Amazon's Elastic MapReduce service, especially if you already have an AWS/EC2 account. The reasons are:
EMR comes with a working Hadoop/HBase cluster "out of the box" - you don't need to tune anything to get Hadoop/HBase working. It Just Works(TM).
Amazon EC2's networking is quite different from what you are likely used to. It has, AFAIK, a 1-to-1 NAT where the node sees its own private IP address, but it connects to the outside world on a public IP. When you are manually building a cluster, this causes problems - even using software like Apache Whirr or BigTop specifically for EC2.
An AMI alone is not likely to help you get a Hadoop or HBase cluster up and running - if you want to run a Hadoop/HBase cluster, you will likely have to spend time tweaking the networking settings etc.
To my knowledge there isn't, but you should be able to easily deploy on EC2 using Apache Whirr which is a very good alternative.
Here is a good tutorial to do this with Whirr, as the tutorial says you should be able to do this in minutes !
The key is creating a recipe like this:
whirr.cluster-name=hbase
whirr.instance-templates=1 zk+nn+jt+hbase-master,5 dn+tt+hbase-regionserver
whirr.provider=ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.hardware-id=c1.xlarge
whirr.image-id=us-east-1/ami-da0cf8b3
whirr.location-id=us-east-1
You will then be able to launch your cluster with:
bin/whirr launch-cluster --config hbase-ec2.properties