We are working on Apache storm with kafka in AWS cluster.
As a developer i want to have a local environment setup to debug the code.
But i checked the prerequisites of Horton sandbox,it requires above 8GB RAM.So we thought to have debugging environment using AWS itself.
Let me know how to do debugging in AWS ot is there any best way to have sandbox with less ram or any another best procedure to follow.
Thanks for your help in advance.
Use Vagrant to manage your EC2 instances, so you can always get the nominated environment for developing
Here is the open source code:
https://github.com/mitchellh/vagrant-aws
The README gives the detail on how to set it with Vagrant.
Related
I have historically used a lot of manual chaining to get a CI pipeline in place for microservice development so am excited to try Fabric8 as it seems that it will make life a lot easier. Running into some early issues though.
I did manage to get Fabric8 running locally but want to get things running on AWS so I can present a more real world flow to stakeholders. Following the notes on this page Fabric8 on AWS I was able to get a 3 server cluster running using Stackpoint. But, I cannot connect to that cluster to be able to start administering the services. The page references this link (http://fabric8.default.replace.me.io) but it is not working for me. Tried hitting each of the AWS instances by public IP but that failed also. What would be my next steps here?
yeah the getting started guides don't really explain this in great deal. There's a similar issue on the fabric8 issue tracker that we've tried to help answer how to access the console
TL;DR using the AWS loadbalancer can add expense so we deploy an NGINX reverse proxy so you can set up a wildcard DNS. We use and recommend cloudflare for that as its free for this type of use and fast to setup.
We also created a blog to explain the different options how to access apps on kubernetes
Hope that helps!
We are looking for the possibility of an automation script which we can give how many master and data nodes we need and it would configure a cluster. Probably giving the credentials in a properties file.
Currently our approach is to login to the console and configure the Hadoop cluster. It would be great if there could be an automated way around it.
I've seen this done very nicely using Foreman, Chef, and Ambari Blueprints. Foreman was used to provision the VMs, Chef scripts were used to install Ambari, configure the Ambari blueprint, and to create the cluster using the Blueprint.
As a part of my college project, I would like to modify Hadoop's source code. However, the problem is that I would need atleast 20 systems to test it. Is it possible to setup this modified version of Hadoop in public clouds such as Google Cloud platform or Amazon Services?Can you give me an idea on the procedure to follow?I could only find information about setting up the original Hadoop versions in the public cloud set up. I couldn't find any information that is relevant to my case.Please do help me out.
Amazon offers elastic mapreduce. But as you correctly pointed out you will not be able to deploy your version of hadoop there.
But you still can use Amazon or Google cloud to just get the base linux servers and install your hadoop on it. It is just a longer process but not different from any other hadoop installation if you have done it before.
I would like to test out Hadoop & HBase in Amazon EC2, but I am not sure how complicate it is. Is there a stable community AMI that has Hadoop & HBase installed? I am thinking of something like bioconductor AMI
Thank you.
I highly recommend using Amazon's Elastic MapReduce service, especially if you already have an AWS/EC2 account. The reasons are:
EMR comes with a working Hadoop/HBase cluster "out of the box" - you don't need to tune anything to get Hadoop/HBase working. It Just Works(TM).
Amazon EC2's networking is quite different from what you are likely used to. It has, AFAIK, a 1-to-1 NAT where the node sees its own private IP address, but it connects to the outside world on a public IP. When you are manually building a cluster, this causes problems - even using software like Apache Whirr or BigTop specifically for EC2.
An AMI alone is not likely to help you get a Hadoop or HBase cluster up and running - if you want to run a Hadoop/HBase cluster, you will likely have to spend time tweaking the networking settings etc.
To my knowledge there isn't, but you should be able to easily deploy on EC2 using Apache Whirr which is a very good alternative.
Here is a good tutorial to do this with Whirr, as the tutorial says you should be able to do this in minutes !
The key is creating a recipe like this:
whirr.cluster-name=hbase
whirr.instance-templates=1 zk+nn+jt+hbase-master,5 dn+tt+hbase-regionserver
whirr.provider=ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.hardware-id=c1.xlarge
whirr.image-id=us-east-1/ami-da0cf8b3
whirr.location-id=us-east-1
You will then be able to launch your cluster with:
bin/whirr launch-cluster --config hbase-ec2.properties
As I want to install Jenkins (ex-Hudson) to operate my continuous integration processes on AWS Beanstalk, I need a custom AMI because some parameters in Tomcat & Linux have to be changed for Jenkins
I run the process of installing and customizing the instance started initially by Beanstalk until the end and Jenkins works like a charm on it.
But, what I can't do is reuse the AMI that I generated at the end of my customization: the health check done by BeansTalk doesn't see the EC2 instance although Beanstalk started it and it works fine.
In order to understand my issue, I reduced my failing process to the following:
a) I create a new BT application / environment based on sample provided by Amazon (only parameter that I had is a keypair to SSH my EC2 instance)
b) when the EC2 instance is started, I use the EC2 to flash the AMI
c) I modify the BT env config by changing the original AWS Ami (id: 100fff79 - Tomcat 6 64 bits) by the 1 that I genrated in (b)
d) the BT rebuilds when I change the ami id
e) the rebuild restarts the EC2 instance.
f) It starts fine (can ssh to it) but the health checking fails and my env turns to red status.
Can somebody replicate this process and tell me what I am doing wrong ?
(I would like to use the AMI of (b) as starting point for my Jenkins customization.?
Additional info that I can provide:
when ssh-ing to the EC2 instance, a grep for apache, java, thin & bluepilld as described at bottom of https://forums.aws.amazon.com/thread.jspa?threadID=59027&tstart=25 shows that the 4 expected processes disappeared. Hence, the failure.
Please, help !
regards
didier
will answer my own question: the right way to obtain a working customized ami for Beanstalk is not to try to flash a running instance launched by Beanstalk but rather start the template ami for Beanstalk (ami-100fff79 for Tomcat 6 64 bits in my case) from EC2 console and customize it from there, flash it and you're done.
You can then "edit configuration" for your BT environment by changing the ami to the new one and it works fine.
regards
didier
If you give more details, this is a feature I'm planning for version 0.3.0 of Beanstalker, my set of Maven plugins for automating maven deployments to Elastic Beanstalk and Elastic MapReduce. It is available at http://beanstalker.ingenieux.com.br/
Actually, the placeholders are there, but I haven't still done full testing of that. Are you willing to try and give help and advice?
You should be able to create a customized AMI from a running instance as long as you delete /opt/elasticbeanstalk/srv/hostmanager/db/hostmanager.db on the instance before building the new AMI. I keep seeing people say "it can't be done, you need to start a clean instance outside of Elastic Beanstalk" and that's bunk. I've done it.
A full write-up of what I've done to customize my install is here: http://stormerider.com/blog/2012/08/16/building-an-ubuntu-ami-with-elastic-beanstalk-support/ -- some of it may not apply to you, some of it may.