What are appropriate scenarios to use userdata, cloud init directives and metadata - amazon-ec2

Please may I have some advice on when to use these settings appropriately? userdata, metadata and cloud-init directives all seem to accomplish the same goal.
Following the AWS documentation, when using userdata I can set up a lamp server in this manner.
#!/bin/bash
yum update -y
amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2
yum install -y httpd mariadb-server
systemctl start httpd
systemctl enable httpd
usermod -a -G apache ec2-user
chown -R ec2-user:apache /var/www
chmod 2775 /var/www
find /var/www -type d -exec chmod 2775 {} \;
find /var/www -type f -exec chmod 0664 {} \;
echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php
Nice and dandy however I can also use cloud-init directives which for me already looks much cleaner.
#cloud-config
repo_update: true
repo_upgrade: all
packages:
- httpd
- mariadb-server
runcmd:
- [ sh, -c, "amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2" ]
- systemctl start httpd
- sudo systemctl enable httpd
- [ sh, -c, "usermod -a -G apache ec2-user" ]
- [ sh, -c, "chown -R ec2-user:apache /var/www" ]
- chmod 2775 /var/www
- [ find, /var/www, -type, d, -exec, chmod, 2775, {}, \; ]
- [ find, /var/www, -type, f, -exec, chmod, 0664, {}, \; ]
- [ sh, -c, 'echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php' ]
Then we have Metadata values in Cloudformation which, to be honest seem more long-winded than cloud-init directives. It goes something like
UserData:
Fn::Base64:
!Sub |
#!/bin/bash -xe
yum update -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource AmazonLinuxInstance --region ${AWS::Region}
systemctl start httpd
Metadata:
AWS::CloudFormation::Init:
config:
packages:
yum:
httpd: []
mariadb-server: []
php: []
php-mysql: []
Tags:
- Key: Project
Value: Autoscale
What are appropriate scenarios to use each of these?

In my view, the main reason you would consider using cfn-init and associated metadata is when you couple it with cfn-hup.
One problem with UserData is that its updates a template do not lead to updates in your instances. This can be problematic if you for example, want to change some config setup of your httpd in your UserData.
The above issue can be overcome when you use cfn-init and cfn-hup. And for many, the ability to update the metadata, and subsequently, the processes on the instance is a major advantage over UserData.

Related

The authenticity of host 'github.com (192.30.253.113)' can't be established passing "yes" with a bash script

I'm working on a personal project that requires me to do some bash scripting. I am currently trying to write a script that pulls from my private git repo. At the moment I am able to spin up my instance and install all my packages through a script. But when it comes to pulling from my private repo I get The authenticity of host 'github.com (192.30.253.113)' can't be established
I am trying to figure out a way to pass "yes" with my script. I know this is very bad practice but for my current use case, I'm not too concerned about security.
running this ssh-keyscan github.com >>~/.ssh/known_hosts command manual works, but when I put this in my script it does not seem to work.
Any help would be greatly appreciated
My script:
echo "update install -start"
sudo yum -y update
sudo amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2
sudo yum install -y httpd mariadb-server
sudo yum install -y git
sudo systemctl start httpd
echo "end"
#file premissions
sudo usermod -a -G apache ec2-user
sudo chown -R ec2-user:apache /var/www
sudo chmod 2775 /var/www && find /var/www -type d -exec sudo chmod 2775 {} \;
#pulling from my git repo
ssh-keyscan github.com >>~/.ssh/known_hosts
cd ../../var/www/html/
git clone git#github.com:jackbourkemckenna/testrepo

docker container nginx start but keeps hanging

my dockerfile:
FROM AWS_ECR_IMAGE
RUN apt-get update && apt-get install -y \
cron \
python-dev \
git \
zlib1g-dev \
libffi-dev \
libssl-dev \
autotools-dev \
automake \
libbz2-dev \
libaio-dev \
libsasl2-dev \
python-pip
RUN pip install boto boto3 awscli
# Install Nginx.
RUN \
add-apt-repository -y ppa:nginx/stable && \
apt-get update && \
apt-get install -y nginx && \
rm -rf /var/lib/apt/lists/* && \
echo "\ndaemon off;" >> /etc/nginx/nginx.conf && \
chown -R www-data:www-data /var/lib/nginx
# Define mountable directories.
VOLUME ["/etc/nginx/sites-enabled", "/etc/nginx/certs", "/etc/nginx/conf.d", "/var/log/nginx", "/var/www/html"]
# Define working directory.
WORKDIR /etc/nginx
# Define default command.
CMD ["nginx"]
COPY nginx_conf /etc/nginx/sites-available/default
# Start service, replace server name, update web ui
COPY main.sh /opt/annotation-pipeline-docs/main.sh
RUN chmod 0755 /opt/annotation-pipeline-docs/main.sh
ENTRYPOINT [ "sh", "-c", "/opt/annotation-pipeline-docs/main.sh" ]
# Expose ports.
EXPOSE 80
And my entrypoint bash file (I need to update the server name first when the container runs) is:
#!/bin/bash -e
/usr/local/bin/aws s3 sync s3://${S3_Bucket}/docs/${ENVIRONMENT}/HEAD/ /var/www/html/
if [ "$ENVIRONMENT" == "prod" ]
then
sed -i.bak "s/REPLACE_ME/example.com/g" /etc/nginx/sites-available/default
else
sed -i.bak "s/REPLACE_ME/example-$ENVIRONMENT.com/g" /etc/nginx/sites-available/default
fi
nginx
while true; do
sleep 60
echo "s3 sync again:"
/usr/local/bin/aws s3 sync s3://${S3_Bucket}/docs/${ENVIRONMENT}/HEAD/ /var/www/html/
done
The issue is when
nginx
runs, it will hanging forever in the terminal:
and the while loop will never get called. Anyone know why is hanging and how to resolve it? Please help, tks in advanced.
The reason for my issue is waiting for the traffic, the while loop will never get called until Nginx start free the bash. However, Nginx will be running in the foreground and not release the focus.
The solution I tried is instead of letting Nginx running as a foreground service, I changed it run in the background. Since this is the only service in my container, should have no problem to do it.
the code changed is simply removed below line in my dockerfile:
echo "\ndaemon off;" >> /etc/nginx/nginx.conf
which will make Nginx as a foreground service

Creating a pod in kubernetes which can run gnu screen

I am creating a long-lived jump to run inside of my kubernetes cluster. It uses an EBS volume for the home folder, holds important copies of my code, and gives me fast access for routine behavior. The problem is that I can't use GNU screen to create similarly long-lived sessions.
Here's my Dockerfile:
FROM ubuntu:zesty
ENV KUBECTL_VERSION=v1.7.6
RUN apt-get update && \
apt-get install -y \
htop vim sysstat \
build-essential make \
ruby ruby-dev rake \
postgresql-client libpq-dev \
curl wget \
python python-pip && \
pip install awscli && \
gem install --no-rdoc --no-ri bundler
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$KUBECTL_VERSION/bin/linux/amd64/kubectl && \
chmod a+x kubectl && \
mv kubectl /usr/local/bin/kubectl
ADD dotfiles /root-dotfiles
ADD code /root-code
ADD docker-entrypoint.sh /docker-entrypoint.sh
And here's how I deploy it:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: doit
purpse: jumpbox
name: doit
namespace: default
spec:
replicas: 1
revisionHistoryLimit: 2
selector:
matchLabels:
app: doit
purpose: jumpbox
template:
metadata:
labels:
app: doit
purpose: jumpbox
spec:
containers:
- image: 123.dkr.ecr.eu-central-1.amazonaws.com/doit:latest
imagePullPolicy: Always
name: doit
command: ["sleep", "infinity"]
workingDir: /root
env:
- name: TERM
value: xterm
volumeMounts:
- mountPath: /root
subPath: root-homedir
name: doit-home
volumes:
- name: doit-home
persistentVolumeClaim:
claimName: doit-home
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 1
But when I kubectl exec in to the container and try creating a screen:
root#doit-2561276907-kl2h6:~# screen -S asdf
Cannot open your terminal '' - please check.
I can work around this by doing:
root#doit-2561276907-kl2h6:~# script /dev/null
Script started, file is /dev/null
# bash
root#doit-2561276907-kl2h6:~# screen -S asdf
# now inside of the screen
Also, here's how I connect to the Pod:
function doit {
doit_pods=$(kubectl get pods -l 'app==doit' -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
kubectl exec -it $doit_pods bash
}
So under the hood I am doing kubectl exec -it.
But I don't want all the users of this jumpbox instance to have to run script and switch back to bash. How can I create Pod which is already configured properly for screen?
I tried this with centos image it works, need to check the ubuntu image.
kubectl run -it screentest --image=centos -- bash
kubectl exec -it screentest-cbd49447f-286wq -- bash
yum -y install screen
screen
Tested this in ubuntu also, it works for me.
kubectl run -it ubuntuest --image=ubuntu -- bash
apt-get update -qq && apt-get install screen -y
screen
also tried with kubectl exec -it ubuntuest-78df75fbb-9sk6f -- bash it works.

Docker centos7 systemctl deos not work : Failed to connect D-bus

I am trying to run elasticsearch on docker.
My features like below
host system : OSX 10.12.5
docker : 17.05.0-ce
docker operating image : centos:latest
I was following this article, but it stuck with systemctl daemon-reload.
I found CentOS official respond about this D-bus bug, but when I ran docker run command it shows the message below.
[!!!!!!] Failed to mount API filesystems, freezing.
How could I solve this problem?
FYI, Here is Dockerfile what I build image
FROM centos
MAINTAINER juneyoung <juneyoung#hanmail.net>
ARG u=elastic
ARG uid=1000
ARG g=elastic
ARG gid=1000
ARG p=elastic
# add USER
RUN groupadd -g ${gid} ${g}
RUN useradd -d /home/${u} -u ${uid} -g ${g} -s /bin/bash ${u}
# systemctl settings from official Centos github
# https://github.com/docker-library/docs/tree/master/centos#systemd-integration
ENV container docker
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ "/sys/fs/cgroup" ]
# yum settings
RUN yum -y update
RUN yum -y install java-1.8.0-openjdk.x86_64
ENV JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64/jre/
# install wget
RUN yum install -y wget
# install net-tools : netstat, ifconfig
RUN yum install -y net-tools
# Elasticsearch install
ENV ELASTIC_VERSION=5.4.0
RUN rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
RUN wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ELASTIC_VERSION}.rpm
RUN rpm -ivh elasticsearch-${ELASTIC_VERSION}.rpm
CMD ["/usr/sbin/init"]
and I have ran with command
docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name=elastic2 elastic2
First, thanks to #Robert.
I did not think it that way.
All I have to do is just edit my CMD command.
Change that to
CMD["elasticsearch"]
However, have to some chores to access from the browser.
refer this elasticsearch forum post.
You could follow the commands for a systemd-enabled OS if you would replace the normal systemctl command. That's how I do install elasticsearch in a centos docker container.
See "docker-systemctl-replacement" for the details.

How to provision software using Vagrant without sudo

I'm trying to setup Vagrant virtual machines to support my learning through Seven Databases in Seven Weeks. I'm provisioning software using basic shell scripts which performs appropriate actions within a sudo environment. However, I'm using the vagrant user to run the tutorials, and would like the provisioning to install the appropriate node / NPM modules as Vagrant, rather than through sudo.
My current npm command is the last line in this provisioning script, but the module is unavailable when vagrant tried to execute node scripts.
apt-get update
apt-get -y install build-essential
apt-get -y install tcl8.5
wget http://redis.googlecode.com/files/redis-2.6.0-rc3.tar.gz
tar xzf redis-2.6.0-rc3.tar.gz
cd redis-2.6.0-rc3
make
make install
make test
mkdir /etc/redis
mv redis.conf /etc/redis/redis.conf
sed -i.bak 's/127.0.0.1/0.0.0.0/g' /etc/redis/redis.conf
sed -i.bak 's/daemonize no/daemonize yes/g' /etc/redis/redis.conf
sed -i.bak 's/dir .\//dir \/var\/lib\/redis/g' /etc/redis/redis.conf
cd src/
wget https://raw.github.com/gist/1053791/880a4a046e06028e160055406d02bdc7c57f3615/redis-server
mv redis-server.1 /etc/init.d/redis-server
mv redis-cli /etc/init.d/redis-cli
chmod +x /etc/init.d/redis-server
sed -i.bak 's/DAEMON=\/usr\/bin\/redis-server/DAEMON=\/usr\/local\/bin\/redis-server/g' /etc/init.d/redis-server
useradd redis
mkdir -p /var/lib/redis
mkdir -p /var/log/redis
chown redis.redis /var/lib/redis
chown redis.redis /var/log/redis
update-rc.d redis-server defaults
/etc/init.d/redis-server start
cd /etc/init.d/
echo ./redis-cli
echo http://blog.hemantthorat.com/install-redis-2-6-on-ubuntu/
apt-get -y install python-software-properties python g++ make
add-apt-repository -y ppa:chris-lea/node.js
apt-get update
apt-get -y install nodejs
npm install hiredis redis csv
Simply set privileged to false in your VagrantFile like this:
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
...
config.vm.provision :shell, privileged: false, path: "script.sh"
...
end
The shell provision runs as the root user. If you with to run as the vagrant user, you can do something like this:
sudo -u vagrant npm install hiredis redis
..or for multiple lines:
sudo -u vagrant << EOF
[...]
npm install hiredis
npm install redis
EOF
Maybe use npm install -g to install it globally in the vm?
sed -i 's/.*requiretty$/Defaults !requiretty/' /etc/sudoers

Resources