I know there are 100+ articles on this topic already, but it does not look like it solves my problem (which still is a very basic and simple).
I am deploying EC2 amazon linux instance via terraform using the following user-data file:
echo "Changing Hostname"
echo "preserve_hostname: true" >> /etc/cloud/cloud.cfg
hostname "${var.prefix}${count.index+1}"
hostnamectl set-hostname --static "${var.prefix}${count.index+1}.localdomain"
echo "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ${var.prefix}${count.index+1}.localdomain" > /etc/hosts
echo "${var.prefix}${count.index+1}" > /etc/hostname
echo "Rebooting"
reboot
After reboot when i log in only the content of /etc/cloud/cloud.cfg is preserved, but the hostname is override by cloud-init. Why ?
When at this stage i change hostname manually and reboot:
hostnamectl set-hostname --static testhostname
reboot
After the reboot i have the correct hostname. Why i do not have the correct hostname after the first reboot ? (so how to set it up correctly from user-data file and why it does not work ?)
UPDATE: ok it looks like i can not use in user-data file terraform variables from ec2 block like "count" - trying to find a workaround....
UPDATE2: looks like i might be forced to use templatefiles like Terraform: How can I pass variables to user_data init script, but to pass count value i would have to build a complex variables/configuration with iteration, there need to be a nice and clean solution here....
UPDATE3: looking for a clean solution without templates, because those are not supported on Mac os M1 platforms :( (https://discuss.hashicorp.com/t/template-v2-2-0-does-not-have-a-package-available-mac-m1/35099)
Thanks,
Related
In a nutshell, after deleting then recreating new global ssh keys on a managed host as part of an ansible play, the shared ssh keys between the controller and the host break. I would like to know a superior method to "fix" this issue and regain the original ssh key trust using ansible itself. Unfortunately this will require some explanation.
Basically as a start, right now, I don't have ansible set up when a new image is deployed. To remedy that, I have created a bash script, utilizing expect which nicely and neatly does 2 things on that new managed host:
Creates an ansible account with appropriate sudo permissions
Creates an ssh key pair between the controller and the controller and the managed host.
That's it, and that's all, however it does require manual input at this time as to the IP of the host to be run on. We now have a desired state from which ansible works well via ssh. However it seems cumbersome at 328 lines of code to check and do this procedure, more on this later.
The issue starts, due to the fact that the host/server is deployed from an image, there is a need to recreate the global keys on each so that they do not have the same set. The fix for this part of that issue is a simple 2 steps:
Find and delete all ^ssh_host_. files in the directory /etc/ssh/
Run the command: /usr/bin/ssh-keygen -A to generate new global ssh keys.
We however now have a problem, once the current ssh connection is broken to the managed host, we can no longer connect to our managed hosts as our known_hosts file on the controller now have keys that don't match. If you do nothing else, you get a prompt again to verify the remote key as it has "changed" and you can't continue until you do. (Stopping all playbooks from functioning) OR if you try to clear the IP out of the known_hosts file on the controller and put it back in, you get the lovely below message:
"changed": false,
"msg": "Failed to connect to the host via ssh: ###########################################################\r\n# WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ***SNIP*** You can use following command to remove the offending key:\r\nssh-keygen -R 10.200.5.4 -f /home/ansible/.ssh/known_hosts\r\nECDSA host key for 10.200.5.4 has changed and you have requested strict checking.\r\nHost key verification failed.",
"unreachable": true
So now I have an issue, and there must be a few commands which I can utilize with ssh-keygen, and/or ssh-keyscan to fix this mess cleanly. However for the life of me I can't figure it out. My only recourse now is to re-run the bash script which initially sets this all up, and replace everything on the controller/host sshkey wise. This seems like overkill, I can't possibly believe that is necessary.
My only hope now is that someone else has an idea how to solve this cleanly and permanently without manual intervention. Otherwise, the only thing I can do is set the ansible_ssh_common_args: "-o StrictHostKeyChecking=no"fact and run the commands my script does but only in playbook form. I can't believe there aren't any modules which can accomplish this. I tried the known_host module, but either I don't know how to use it properly, or it doesn't have this functionality. (Also it has the annoying property of changing my known_hosts file to root ownership, which I must then change back.)
If anyone can help that would be fantastic! Thanks in advance!
The below is not strictly needed as it's extra text clogging up the works, but it does illustrate how the bash script fixes this issue and maybe give some insight on a better solution:
In short, it generates an ssh public and private key, attaches the hostname to them, creates an ssh config identity file using a heredocs population method, puts them in the proper spots, and then copies the public key over to mangaged host in question.
The code snipits are below to show how this is accomplished. This is not the entire script just relevant parts:
#HOMEDIR is /home/ansible This host is the IP of managed host in the run.
#THISHOST is the IP of the managed host in question. Yes, we ONLY use IP's, there is no DNS.
cd "$HOMEDIR"
rm -f $HOMEDIR/.ssh/id_rsa
ssh-keygen -t rsa -f "$HOMEDIR"/.ssh/id_rsa -q -P ""
sudo mkdir -p "$HOMEDIR"/.ssh/rsa_inventory && sudo chown ansible:users "$HOMEDIR"/.ssh/rsa_inventory
cp -p "$HOMEDIR"/.ssh/id_rsa "$HOMEDIR"/.ssh/rsa_inventory/$THISHOST-id_rsa
cp -p "$HOMEDIR"/.ssh/id_rsa.pub "$HOMEDIR"/.ssh/rsa_inventory/$THISHOST-id_rsa.pub
#Heredocs implementation of the ssh config identity file:
cat <<EOT >> /home/ansible/.ssh/config
Host $THISHOST $THISHOST
HostName $THISHOST
IdentityFile ~/.ssh/rsa_inventory/${THISHOST}-id_rsa
User ansible
EOT
#Define the variable earlier before the expect script is run so it makes sense in next snipit:
ssh_key=$( cat "$HOMEDIR"/.ssh/id_rsa.pub )
#Snipit in except script where it echos over the public ssh key to the managed host from the controller.
send "sudo echo '"$ssh_key"' >> /home/ansible/.ssh/authorized_keys\n"
expect -re {:~> *$}
send "sudo chmod 644 /home/ansible/.ssh/authorized_keys\n"
expect -re {:~> *$}
#etc etc, so on and so forth properly setting attributes on this file. ```
Now things work with passwordless ssh as they should. Until they are re-ruined by the global ssh key replacement.
service:
volumes:
- ~/.gitconfig:/root/.gitconfig
- ~/.pgpass:/root/.pgpass
- $SSH_AUTH_SOCK:/ssh-agent
- /etc/hosts:/etc/hosts
environment:
- SSH_AUTH_SOCK=/ssh-agent
At running i try to modify /etc/hosts inside of container
#!/bin/sh
alias_host_name="db"
container_ip=$(hostname -I|awk -F\ '{ print $(NF) }')
echo "Set route: $alias_host_name -> $container_ip"
grep -q $alias_host_name /etc/hosts
hosts_have_alias=$(echo $?)
if [ $hosts_have_alias -ne 0 ]; then
# no data for alias_host_name
echo "$container_ip $alias_host_name" >> /etc/hosts
else
# data for alias_host_name already exists
cp /etc/hosts ~/hosts.new
reg_exp_ip="((1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])"
sed -ri "s/$reg_exp_ip \t$alias_host_name/$container_ip $alias_host_name/" ~/hosts.new
sed -ri "s/$reg_exp_ip $alias_host_name/$container_ip $alias_host_name/" ~/hosts.new
cat ~/hosts.new > /etc/hosts
fi
so i expect that /etc/hosts will have new line
it works fine at linux
thx for help!
Docker manages the /etc/hosts file for you. You can't provide it in an image or bind-mount it. This is also true of /etc/resolv.conf, and it's part of how Docker provides a network environment for a container.
For what you're showing you don't need /etc/hosts at all. If you're running the containers in Docker Compose, it automatically provides DNS resolution between containers so a host name db refers to the container that's declared as db: in the docker-compose.yml file. This is provided out-of-the-box; you don't need to do anything at all for this functionality to work.
If you need to give an alternate name to a container, you can declare aliases: for it in the docker-compose.yml file. It'd be a little unusual to need this, though.
Heavily relying on /etc/hosts usually isn't a best practice. Keeping hosts files in sync across multiple environments is difficult, and in your example where you're shadowing a host name that already exists, the behavior can be different in different programs (depending on whether it uses the system resolver or only true DNS). If you must, though, you can specify extra_hosts: in the docker-compose.yml file to have Docker set it up for you.
Whats a bulletproof way to determine if I am running inside a vagrant machine?
Guest OS is Debian Linux, though if there are indicators per-os that would be great to have documented as well.
AFAIK, there isn't a way outside of your own customizations. One idea that comes to mind is to touch a file that you then reference from your apps. Add something like config.vm.provision "shell", inline: "touch /etc/is_vagrant_vm" to the bottom of your Vagrantfile and base your conditionals around the existence of that file.
Provisioning a file that you can check the existence of seems like the most reliable way to handle this.
Another option if you don't want to write files to the box would be to check for the existence of the vagrant user itself:
grep -q '^vagrant:' /etc/passwd && echo 'Vagrant environment: true'
This is not foolproof as others have indicated, and it is possible (although uncommon) to have a vagrant box that uses a different user to connect as.
Checking the user's existence also would not work reliably if you have machines in your environment with a user account called vagrant that are not actually vagrant boxes, but that would also be fairly uncommon.
I don't know if there is any bulletproof way for this, but one thing I often do is to config my vagrant environment's shell UI to be different from the shell UI of my host machine. This way I can tell the difference at first glance. It also helps if you want to distinguish among multiple vagrant environments.
To customize the shell UI, oh-my-zsh will come in handy.
Assuming you don't run a vagrant instance in your vagrant instance:
if which vagrant > /dev/null 2>&1; then
echo "This is most likely the host"
fi
You can use facter
facter | grep vagrant
This is how I was able to answer the question with a very small chunk of bash. The -e flag checks if the file exists. Change provision to a file path that makes sense for your build.
#!/usr/bin/env bash
user=$USER
provision=/vagrant/Vagrantfile
if [ -e $provision ]
then
echo "Vagrantfile found..."
echo "Setting $user to vagrant..."
user=vagrant
fi
# Example usage
cd /home/$user
Tested on "centos/7"
In vagrant your are in control of which IP you are running against, so you could change it to for example 192.168.0.10
Then:
curl localhost:8080/health => in local e.g spring-boot
curl 192.168.0.10:8080/health => in vagrant
NOTE: /health assumes spring-boot but you could also use your own implementation of a /health endpoint as well
Is it possible for one to modify files on the host machine during the vagrant up process? For example, adding an entry to the host machine's /etc/hosts file to avoid having to do this manually?
The solution is to use vagrant-hostsupdater
vagrant plugin install vagrant-hostsupdater
This plugin adds an entry to your /etc/hosts file on the host system.
On up and reload commands, it tries to add the information, if its not
already existant in your hosts file. If it needs to be added, you will
be asked for an administrator password, since it uses sudo to edit the
file.
On halt, suspend and destroy, those entries will be removed again.
OK, so now the guy sitting next to you at the coffee shop can most likely ssh to port 2222 (EDIT: changed on newer versions of vagrant, unless you explicitly enable external access) on your computer, login as vagrant with the insecure key, modify your Vagrantfile, since it's mounted read-write and owned by the vagrant user, insert arbitrary ruby code to run in the host environment, and now it looks like they've got root access on the host environment as well. Brilliant.
I hope people run firewalls on their development machines.
EDIT:
So after writing the above, I bugged the author of Vagrant, the default has been changed so that port 2222 is not open by default on the external interface. Big improvement (though still something to be careful of, since external access is often opened up for various reasons).
So, having put in effort to get the situation fixed since making this comment, I'm now getting down votes, apparently because the comment is out of date. Damn. It was correct when written.
EDIT:
In response to Steve Buzonas, the point is that if there's any likelhihood of the virtual machine being compromised then giving the vagrant up process elevated permissions represents a serious risk to the security of the host environment, and also being able to modify the /etc/hosts environment file is dangerous, even without general root access. As I've pointed out, vagrant's approach to keeping the VM secure is not particularly rigorous.
I don't want to depend on some plug in to vagrant. It should be standard feature in Vagrant!!!! Untill then I use a shell script to propagate VM's in my cluster of new VMs. The key lines are :
# Obtain the hostkey based on the IP-address and add it to the known_host list
ssh-keyscan -t ecdsa ${START}.${OFFSET} >> /home/vagrant/.ssh/known_hosts
# obtain the hostname, because you might not know it yet, with the IP address:
EXTERNAL_HOSTNAME=`ssh ${START}'.'${OFFSET} 'hostname'`
# obtain the key ot the new other VM based on hostname and also add to known_hosts
ssh-keyscan -t ecdsa ${EXTERNAL_HOSTNAME} >> /home/vagrant/.ssh/known_hosts
# so now you have the IP address and the corresponding hostname
# add to /etc/hosts without being asked for "yes/no"
echo ${START}'.'${OFFSET}' '${EXTERNAL_HOSTNAME} >> /etc/hosts
Where IPADRRESS is the IP address of the master VM in the cluster with several slave node VM's with succeedding ip-addresses. (IPADDRESS=IPADDRESS + 1 untill no successfull ping)
IPADDRESS=`ip addr show eth1 | grep 'inet ' | cut -d ' ' -f 6 | cut -d '/' -f1`
START=`echo ${IPADDRESS} | cut -d '.' -f1,2,3`
OFFSET=`echo ${IPADDRESS} | cut -d '.' -f4`
And then I loop trough the next IP addresses until no more succesfull pings.
I do not want to hardcode anything (ip-address or hostname), but to find out itself.
Resulting /etc/hosts file (after
sort /etc/hosts | uniq > /tmp/hosts.uniq && sudo sh -c 'mv /tmp/hosts.uniq /etc/hosts'
:
[vagrant#master ~]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
127.0.0.1 master.RHEL70.local master
192.168.1.50 master.RHEL70.local
192.168.1.51 node01.RHEL70.local
192.168.1.52 node02.RHEL70.local
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Previously I didn't know how to vagrant edit my etc/host file. But when i reinstalled window and vagrant, this feature disappeared.
I'm running a couple of standard Fedora instances on EC2. I feel the public hostnames of the instances assigned by Amazon are too weird and hard to remember. I'd like to change them to something short (like red/blue/green/etc).
Is there any draw back in doing this? And how do I set it up such that it persists after reboots?
Thanks.
Before you get started, try running hostname and hostname --fqdn and take note of what the responses are.
You can edit /etc/hostname and set a hostname, which will stick around after rebooting. You can force the hostname to be "reloaded" by using hostname -F /etc/hostname to read that value into the hostname. The bash prompt will change after you logout and login.
warning / note:
Yes, it is nice to have the hostname in the bash prompt set to something more useful than ip-123-123-123-123 but I've decided to leave mine (at least for now) because it seems like a lot of things really count on having the hostname on ec2 instances set in a standard way. After editing /etc/hostname and changing the hostname to webserver a lot of the services seems to fail because the hostname would not resolve, and apache wouldn't start. Next I edited /etc/hosts and added in
127.0.0.1 webserver
as the second line. Apache would then start but complained that it couldn't find the FQDN. I confirmed that running hostname --fqdn no longer worked.
Next I consulted man hostname and learned that while you can set the hostname it appears that the FQDN is what is returned via a DNS lookup.
THE FQDN
You can't change the FQDN (as returned by hostname --fqdn) or the DNS domain name (as returned by dnsdomainname) with this command. The FQDN of the system is the name that the resolver(3) returns for the host name.
Technically: The FQDN is the name getaddrinfo(3) returns for the host name returned by gethostname(2). The DNS domain name is the part after the first dot.
Therefore it depends on the configuration (usually in /etc/host.conf) how you can change it. Usually (if the hosts file is parsed before DNS or NIS) you can change it in /etc/hosts.
I think it might be possible to set the system / fool the system into return the FQDN, something like ip-123-123-123-123.ec2.internal even though the hostname is webserver but at this point it started to seem like more trouble than it was worth, and that for me to have a nicer bash prompt might cause a lot software and configuration problems down the road and so I decided to give up.
I also learned that a lot of amazon ec2 instances use something called cloud-init:
cloud-init is the Ubuntu package that handles early initialization of a cloud instance. It is installed in the Ubuntu Cloud Images and also in the official Ubuntu images available on EC2.
Some of the things it configures are:
setting a default locale
setting hostname
generate ssh private keys
adding ssh keys to user's .ssh/authorized_keys so they can log in
setting up ephemeral mount points
cloud-init's behavior can be configured via user-data. User-data can be given by the user at instance launch time. This is done via the --user-data or --user-data-file argument to ec2-run-instances
I also found this which talks about how the hostname is configured with cloud-init:
On EBS instances, a shutdown and later start would end up with a different IP address.
In the case where the user has not modified /etc/hostname from its original value (seeded by metadata's 'local-hostname'), then cloud-init will again set the hostname and update /etc/hostname.
In the case where the user has modified /etc/hostname, it will remain user managed.
Additionally, if /etc/cloud/cloud.cfg contains 'preserve_hostname' value set to a True value, then /etc/hostname will not ever be touched.
The interesting takeaway is that if you don't change the hostname the cloud-init package will keep it up to date for you.
If someone else has a workaround or can address some of the issues mentioned and help reassure that nothing will break on ec2 instances because of changing the hostname I would be happy to hear it.
Another way is to simply edit ~/.bashrc and prepend PS1 with the nickname of the machine.
Edit: perhaps more correctly, machine-wide, e.g. on the AWS Linux AMI (an example) (paste this into console or add to your arbitrary install .sh):
cat << EOF | sudo tee /etc/profile.d/ps1.sh
if [ "$PS1" ]; then
PS1="[\u#myinst1:\l \t \! \W]\\$ "
fi
EOF
Edit /etc/sysconfig/network as root.
Replace
HOSTNAME=localhost.localdomain
with
HOSTNAME=hostname.DOMAIN_NAME
Then, either reboot or run /etc/init.d/network restart
The server then should report its name as a FQDN.
From this site:
Change the hostname on a running system
On any Linux system you can change its hostname with the command hostname (surprised?)…
Here are some quick usages of the command line hostname:
$> hostname
without any parameter it will output the current hostname of the system.
$> hostname --fqd
it will output the fully qualified domain name (or FQDN) of the system.
$> hostname NEW_NAME
will set the hostname of the system to NEW_NAME.
You can also edit /etc/hostname (at least on Ubuntu).
To make sure it stays after a reboot in AWS, either add the command in /etc/rc.local so it runs when the machine starts.
There's also a way to set the hostname dynamically via USER_DATA:
USER_DATA=`/usr/bin/curl -s http://169.254.169.254/latest/user-data`
HOSTNAME=`echo $USER_DATA`
IPV4=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/public-ipv4`
hostname $HOSTNAME
echo $HOSTNAME > /etc/hostname
To change the system hostname to a public DNS name
Follow this procedure if you already have a public DNS name registered
Open the /etc/sysconfig/network configuration file in your favorite text editor and change the HOSTNAME entry to reflect the fully qualified domain name (such as webserver.mydomain.com).
HOSTNAME=webserver.mydomain.com
Reboot the instance to pick up the new hostname.
[ec2-user ~]$ sudo reboot
Log into your instance and verify that the hostname has been updated. Your prompt should show the new hostname (up to the first ".") and the hostname command should show the fully qualified domain name.
[ec2-user#webserver ~]$ hostname
webserver.mydomain.com
To change the system hostname without a public DNS name
Open the /etc/sysconfig/network configuration file in your favorite text editor and change the HOSTNAME entry to reflect the desired system hostname (such as webserver).
HOSTNAME=webserver.localdomain
Open the /etc/hosts file in your favorite text editor and add an entry beginning with 127.0.1.1 (on DHCP systems) or eth0's address (on static IP systems) to match the example below, substituting your own hostname. (127.0.0.1 should be left as the localhost line.)
127.0.0.1 localhost localhost.localdomain
127.0.1.1 webserver.example.com webserver
Reboot the instance to pick up the new hostname.
[ec2-user ~]$ sudo reboot
Log into your instance and verify that the hostname has been updated. Your prompt should show the new hostname (up to the first ".") and the hostname command should show the fully qualified domain name.
[ec2-user#webserver ~]$ hostname
webserver.localdomain
Note: You can also change the shell prompt without affecting the hostname. Refer to this AWS documentation.
Sure, you can do that if you have your own domain (setup a CNAME to point to the Amazon hostname). Otherwise, you're pretty much stuck with the one they give you (or an Elastic IP, if you set one of those up).
The /etc/rc.local solution worked for me for a basic hostname but does not give me a FQDN.
In my Linux AMI (a snapshot of other instance).. none of the above formula worked. Then, I simply changed HOSTNAME field in file: /etc/init.d/modifyhostname and did a normal reboot.
You will need to do multiple things to set the hostname:
hostname newname - sets the hostname, but is volatile
edit /etc/hostname - sets the hostname for the next reboot
edit /etc/hosts - to keep sudo from complaining
I put these together into a script and uploaded it as a gist:
https://gist.github.com/mnebuerquo/5443532036af8b48995547e2817dba85
sudo hostname *yourdesiredhostnamehere*
sudo /etc/init.d/networking restart
then the hostname is changed. On my server all other services like apache and postfix works. Server is Ubuntu 12.04 LTS