I'm trying to setup DC/OS on AWS with different resource roles (I don't want nodes reserved for Apache Cassandra used by other things). I know in theory how to do it in a plain Mesos that I set up manually (https://support.mesosphere.com/hc/en-us/articles/206474745-How-to-reserve-resources-for-certain-frameworks-in-Mesos-cluster-), but I don't know how to do it in a DC/OS cloud installation. Any advice is greatly appreciated.
DC/OS neither facilitates, nor prohibits services (aka frameworks) to register with a custom role. I'm not sure how to tell Cassandra to register under special role, Jenkins for example exposes it in its configurations (see screenshot below). In short, check with your framework, how to register in a custom role, DC/OS does not restrict this.
If you want to configure role-specific things, like weights or quota, just talk directly to Mesos, which is accessible via /mesos, DC/OS does not prohibit it either.
Related
Kinda newbie on DC/OS so bear with me on this rather trivial question.
I have set up a HA DC/OS cluster.
I want to deploy some container services.
What is the difference between running services
a) By selecting the Services option on the left pane that prompts you with the following options below
b) by installing Marathon from the catalogue and then running service on it
Well, a) will ultimately use the already present Marathon instance, and b) will install a Marathon-on-Marathon instance, which is unnecessary if you just want to run a few containers and/or services.
We are looking for the possibility of an automation script which we can give how many master and data nodes we need and it would configure a cluster. Probably giving the credentials in a properties file.
Currently our approach is to login to the console and configure the Hadoop cluster. It would be great if there could be an automated way around it.
I've seen this done very nicely using Foreman, Chef, and Ambari Blueprints. Foreman was used to provision the VMs, Chef scripts were used to install Ambari, configure the Ambari blueprint, and to create the cluster using the Blueprint.
I am very new to the Amazon AWS services. I was wondering if there is a way to run an instance of EC2 (say, Amazon Linux AMI) and then connect two environments to this instance.
Particularly, I'd like to run a PHP and a Tomcat environment on a single EC2 instance.
The problem is, every time I create a new environment in Elastic Beanstalk, it seems to create a new EC2 instance as well. Am I missing something here?
I'd appreciate any hint on this.
AWS Elastic Beanstalk is designed for deploying your running apps in a way that is designed for scalability from the ground-up. Because of this, Elastic Beanstalk will launch one or more EC2 instances, connect them to an Elastic Load Balancer instance, configure CloudWatch monitoring and Auto Scaling triggers.
Also, because of its fundamental design for scalability, Elastic Beanstalk is designed around a one-app-per-environment model (whereby "environment", I mean one of these EC2 + ELB + CloudWatch + AutoScaling clusters).
Since running two separate web servers with two separate apps (PHP & Java) is not a fundamentally scalable design, it's not a use-case that Elastic Beanstalk is optimized for.
You are free to spin-up a standalone EC2 instance and install whatever you'd like on it, but you're right — git aws.push support has not been made available for standalone EC2 instances. If the git support is important to you, you'll need to weigh the pros and cons of each approach.
I would also like to be able to do this, basically from a cost perspective for demos etc.
For example, a single instance with one PHP app and one Java app. Or, a single instance with two Java apps.
However, from what I have read so far in the Elastic Beanstalk developer guide, I have not found anything explicitly stating that multiple applications per environment is supported (or even, multiple environments per EC2 instance - if that even makes sense).
It makes me wonder if this is a feature that is often requested and planned for the future, or alternatively if the single-app-per-environment model is 'by design' for some reason.
I usually use Munin as monitoring software, but this (as others software I presume) needs an IP to make the ICMP or whatever pings to collect data.
In Amazon EC2 instances are created on the fly, with IP's you don't know.
How can they be monitored ?
I was thinking about using amazon console commands to read the IP's of the instances up, and change the monit configuration file on the fly also , but it can be too complicated ... or not?
Any other solution / suggestion ?
Thank you
I use revealcloud to monitor my amazon instances. You can install it once and create an ami from that systen, or bootstrap the install command if that's your method. Since the install is just one command, it's easy enough to put into the rc.local (or similar). You can then see all the instances in the dashboard or topiew as soon as they boot up.
Our instances are bootstrapped using chef recipes, so it's easier for me to provide IPs/hosts as they (= all members of my cluster) get entered into /etc/hosts on start-up. Generally, it doesn't hurt to use elastic IPs for a master server and allow all connections (in /etc/munin/munin.conf by default).
I'd solve the security 'question' on the security groups level. E.g. allow only instances with a certain security group to connect to the munin-node process (on port 4949). The question which remains is.
E.g., using ec2-authorize you can achieve
ec2-authorize mygroup -o monitorgroup -u <AWS-USER-ID>
This means that all instances with group monitorgroup can access resources on instances with mygroup.
Let me know if this helps!
If your Munin master and nodes are all hosted on EC2 than it's better to use internal hosts like domU-00-00-00-00-00-00.compute-1.internal. because this way you don't have to deal with IP addresses and security groups.
You also have to set this in /etc/munin/munin-node.conf:
allow ^.*$
You can read more about it in Monitoring AWS Ubuntu Instances using Munin
But if your Munin master is not on EC2 your best bet is to attach Elastic IP to your EC2 instance.
When autoscaling my EC2 instances for application, what is the best way to keep every instances in sync?
For example, there are custom settings and application files like below...
Apache httpd.conf
php.ini
PHP source for my application
To get my autoscaling working, all of these must be configured same in each EC2 instances, and I want to know the best practice to sync these elements.
You could use a private AMI which contains scripts that install software or checkout the code from SVN, etc.. The second possibility to use a deployment framework like chef or puppet.
The way this works with Amazon EC2 is that you can pass user-data to each instance -- generally a script of some sort to run commands, e.g. for bootstrapping. As far as I can see CreateLaunchConfiguration allows you to define that as well.
If running this yourself is too much of an obstacle, I'd recommend a service like:
scalarium
rightscale
scalr (also opensource)
They all offer some form of scaling.
HTH