Run deck Automation script - shell

I need a help, I am trying to write bash/shell script which will be placed in Rundeck tool. As my org has more than 10,000 severs Ec2. This is what I am expecting.
script to login into Ec2.
show output of df -h,lsblk & Java version.
Please anyone help me with the script.

You need to configure the EC2 nodes on Rundeck using a model source. To avoid configuring each EC2 node you can use the EC2 Model Source Plugin, take a look at this (check this to learn how to install plugins on Rundeck), after that you can create a job with an inline script step using a node filter pointing to your ec2 nodes.

Related

How to run Ansible play-book command from remote server

I need to install and configure all new system start with auto-scaling in aws as per the requirements , like if it is a app server install nodejs with respective git code for deployment using with Ansible.
How Ansible identify a new system came up and need to do this all configuration.
Here is a guide from ansible docs how to handle autoscaling with Ansible: https://docs.ansible.com/ansible/latest/scenario_guides/guide_aws.html#autoscaling-with-ansible-pull
The problem on this approch is, that you need the whole provisining prozess on startup. This takes much time and is error prone.
A common solution is to build a custom AMI with all infrastructure needed for your service and only deploy your current code to this maschine.
A good tool to build custom AMIs is Packer. A Guide for AWS is available here. https://www.packer.io/docs/builders/amazon.html

Fabric scripts to install Hadoop on a cluster of machines?

I am beginning to install hadoop on a cluster. I have ssh access to these machines and I have already installed fabric on them. I was wondering if someone has already written a fabfile to install and deploy hadoop to a cluster easily.
I found this project [0]; but this is written for deploying over AWS instances. I was looking for something where I can just fill in the IPs of my machines and then execute a set of fab commands to bring up the cluster.
[0] http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide/#ec2-deployment-with-fabric-script
I'm AlexJF, the author of the scripts you linked.
The scripts you reference can also be used outside EC2. You just need to configure, as you requested, the list of hosts and configurations on the top of the fabfile.py. Be sure to set EC2 = False (which just happens to be the default).
You'll then have several useful commands available to you.

Configuring AWS cluster using automation script

We are looking for the possibility of an automation script which we can give how many master and data nodes we need and it would configure a cluster. Probably giving the credentials in a properties file.
Currently our approach is to login to the console and configure the Hadoop cluster. It would be great if there could be an automated way around it.
I've seen this done very nicely using Foreman, Chef, and Ambari Blueprints. Foreman was used to provision the VMs, Chef scripts were used to install Ambari, configure the Ambari blueprint, and to create the cluster using the Blueprint.

Assigning the host name to a hadoop job using weka in AWS

I have been using the wekaDistributedHadoop1.0.4 and wekaDistributedBase1.0.2 packages on my local machine to run some of the basic jobs. There is a field "HDFS host" which must be filled in order to run the jobs. I have been using "localhost" since I have been testing on my local machine and this works fine. I blindly tried using "localhost" when running on AWS EMR but the job failed. What I would like to know is what host name should I be entering into the field so that weka will call on the correct master? Is it the public DNS name which is provided when starting the cluster or is there a method in the API which gets that address for me?
If you want to manually do it.
Create a cluster and keep it alive, you can find info in amazon ec2 instances manage console, in security group elastic mapreduce master/slave. Find it out, login master node and edit conf file and fill with right name.
If you need automatically do it.
Write a shell executed in bootstrap. You can refer to https://serverfault.com/questions/279297/what-is-the-easiest-way-to-get-a-ec2-public-dns-inside-a-running-instance

How to sync my EC2 instance when autoscaling

When autoscaling my EC2 instances for application, what is the best way to keep every instances in sync?
For example, there are custom settings and application files like below...
Apache httpd.conf
php.ini
PHP source for my application
To get my autoscaling working, all of these must be configured same in each EC2 instances, and I want to know the best practice to sync these elements.
You could use a private AMI which contains scripts that install software or checkout the code from SVN, etc.. The second possibility to use a deployment framework like chef or puppet.
The way this works with Amazon EC2 is that you can pass user-data to each instance -- generally a script of some sort to run commands, e.g. for bootstrapping. As far as I can see CreateLaunchConfiguration allows you to define that as well.
If running this yourself is too much of an obstacle, I'd recommend a service like:
scalarium
rightscale
scalr (also opensource)
They all offer some form of scaling.
HTH

Resources