I successfully deployed Kubernetes on AWS using "getting started on AWS ec2 guide" (http://kubernetes.io/v1.0/docs/getting-started-guides/aws.html), but the disk size of all the minions (kubernetes hosts) is 8gb. I would like to increase the disk size, but I haven't found a way to do it.
I can change the VM size by setting MINION_SIZE (e.g. export MINION_SIZE=m3.medium) prior to installing, but the disk size is still 8gb.
From the Kubernetes install instructions for other cloud providers there's an option to set MINION_DISK_SIZE to set the disk size. I tried that with AWS ec2 installation, and the variable is ignored.
I also poked around the config files, but I didn't see anything obvious.
Any suggestions on how to set the disk size for minions when installing Kubernetes on AWS ec2?
I recently stumbled upon the same issue. Have a look at BLOCK_DEVICE_MAPPINGS in kubernetes/cluster/aws/util.sh. You can modify it to have something more appropriate for a EBS-only minion.
For example:
[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":80}}]
AWS docs: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html
I have faced this very issue and tried the current accepted answer but it looks like Kubernetes is changing quite fast what may make this answer also outdated soon.
To this date, I've tested the solution below that might become or not a definitive solution in the future:
There is this PR on Kubernetes' github project that implements an easy way to ignore the SSD storage by setting KUBE_AWS_STORAGE=ebs before running kubernetes/cluster/kube-up.sh.
Hope it is helpful!
Related
I have been trying to spin up a Kubernetes/Fabric8 installation on AWS using Stackpoint as described in this video: https://www.youtube.com/watch?v=lNRpGJTSMKA
My problem is that three of the apps wont start becuase no volumes are available and I cannot see how to resolve those PV requests. For example Gogs is reporting the following error:
Unable to mount volumes for pod "gogs-2568819805-bcw8e_default(03d618b9-7477-11e6-8c6b-0a945216fb91)": timeout expired waiting for volumes to attach/mount for pod "gogs-2568819805-bcw8e"/"default". list of unattached/unmounted volumes=[gogs-data]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "gogs-2568819805-bcw8e"/"default". list of unattached/unmounted volumes=[gogs-data]
I am pretty sure this is very simple but cannot see how to connect the dots here from the various K8, Fabric8 docs. I can create a new EBS volume in AWS easily enough but cannot see how to then update this running stack to attach it to these services. Any help would be greatly appreciated!
Sorry about that, what version of gofabric8 are you using? We're currently adding persistent volume support for the core platform apps although the integration our stackpoint isn't there quite yet. Hopefully soon though.
For now you should be able to disable the PV claims using --pv=false during the deploy. So gofabric8 deploy --pv=false. We'll look at using this as the default until the integration is there and we can leverage AWS persistent volumes
We just shipped functionality that allows you to create and manage AWS volumes for Kubernetes. You get a volume, PV, and claim - just name the claim to be what is required by Fabric8. Eventually, you'll be able to use dynamic volume creation.
I would like to install DSE on m4.* instances, but it seems that the latest ComboAMI doesn't support it. Since it requires a different configuration, as it's not an instance-store backed instance, but rather an ebs-store backed instance and also a HVM instance and not PV instance.
I've started working on a new provisioning script, that basically does what the ComboAMI provisioning script does, but also adds the RAID0 creation for the EBS volumes that I attach per node, and the xfs file system creation.
My question is divided to two parts:
First of all, I was wondering if anyone in DataStax has already done something similar that he can share, since that can be a great reference for anyone that wants to do something similar.
Second of all, has anyone had any experience building a similar AMI on a m4.* instance type? I didn't manage to get something to work with the ComboAMI project.
We are working on Apache storm with kafka in AWS cluster.
As a developer i want to have a local environment setup to debug the code.
But i checked the prerequisites of Horton sandbox,it requires above 8GB RAM.So we thought to have debugging environment using AWS itself.
Let me know how to do debugging in AWS ot is there any best way to have sandbox with less ram or any another best procedure to follow.
Thanks for your help in advance.
Use Vagrant to manage your EC2 instances, so you can always get the nominated environment for developing
Here is the open source code:
https://github.com/mitchellh/vagrant-aws
The README gives the detail on how to set it with Vagrant.
I would like to test out Hadoop & HBase in Amazon EC2, but I am not sure how complicate it is. Is there a stable community AMI that has Hadoop & HBase installed? I am thinking of something like bioconductor AMI
Thank you.
I highly recommend using Amazon's Elastic MapReduce service, especially if you already have an AWS/EC2 account. The reasons are:
EMR comes with a working Hadoop/HBase cluster "out of the box" - you don't need to tune anything to get Hadoop/HBase working. It Just Works(TM).
Amazon EC2's networking is quite different from what you are likely used to. It has, AFAIK, a 1-to-1 NAT where the node sees its own private IP address, but it connects to the outside world on a public IP. When you are manually building a cluster, this causes problems - even using software like Apache Whirr or BigTop specifically for EC2.
An AMI alone is not likely to help you get a Hadoop or HBase cluster up and running - if you want to run a Hadoop/HBase cluster, you will likely have to spend time tweaking the networking settings etc.
To my knowledge there isn't, but you should be able to easily deploy on EC2 using Apache Whirr which is a very good alternative.
Here is a good tutorial to do this with Whirr, as the tutorial says you should be able to do this in minutes !
The key is creating a recipe like this:
whirr.cluster-name=hbase
whirr.instance-templates=1 zk+nn+jt+hbase-master,5 dn+tt+hbase-regionserver
whirr.provider=ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.hardware-id=c1.xlarge
whirr.image-id=us-east-1/ami-da0cf8b3
whirr.location-id=us-east-1
You will then be able to launch your cluster with:
bin/whirr launch-cluster --config hbase-ec2.properties
I couldn't find the answer on EC2 documentation. What is it for? If I launched an EBS backed instance, the root device for the instance would be an EBS volume. If I install a few tools/software on the instance, will those be installed on the root instance by default? Still I guess the question really came from the little understanding of the root device. Any detailed info on that?
Also if I need to launch another EBS backed instance, and also want to have the same copy of the tools/software installed on the earlier instance, how to do that?
EC2 instances have two types of storage -- ephemeral storage, and EBS based storage -- and each instance is given a specific amount of that disk space by default. Each instance has its own allotment of disk space that it can use which is independent of any other instances disk space that you launch. You can also add an EBS volume to your drive for additional storage, but, those volumes can only be attached to one specific host at a time.
If you had two EBS instances, and you wanted to ensure that they're both using the same tools and software, you'd need to enforce that using a configuration management tool, such as puppet, chef, or cfengine.
I agree with what Ian wrote. I would add that the "root device" in EC2 is analogous to the operating system partition in a personal computer. It is where the filesystem of your OS resides.
This is the best answer about 'root device' so far! AWS is still new. And most of the times, the answer is just 'cut and paste' straight from AWS document, which does not help much!
https://web.archive.org/web/20140601070714/https://skeddly.desk.com/customer/portal/articles/1346918-ebs-backed-versus-instance-store
This may have been added since you asked, but I believe this answers the question:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html