kubeadm join can't connect - amazon-ec2

I created a single-node cluster on a Ubuntu 18.04 node on EC2, using kubeadm init. However I am unable to join (unable to connect to the API) from another node.
Note: this is an EC2 instance.
Kubectl is working fine on the master itself.
I used the following command where MASTER_PRIVATE_IP is 172.31.25.111.
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=${MASTER_PRIVATE_IP} --apiserver-cert-extra-sans=${MASTER_PUBLIC_IP}
When I try to join a second node on the same private network to the cluster with kubeadm join it just times out. I can ssh to the master no problem and when performing a netstat on the master I see that it only seems to be listening to port 6443 on ipv6 addresses - why?
I provided the private IPv4 address as the advertise address.
(kubeconfig has htat private ipv4 address of course).
kube-apiserver --authorization-mode=Node,RBAC --advertise-address=172.31.25.111 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
netstat -tulpn | grep -E ":(22|6443)" | grep LISTEN
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::6443 :::* LISTEN -
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=${MASTER_PRIVATE_IP} --apiserver-cert-extra-sans=${MASTER_PUBLIC_IP}
Any ideas?

add security group to Master ec2, for instance this way
Port range:
0 - 6555 or just 6443
source ip
172.31.0.0/16

To add to #Markownikow's answers, in this instance, your issue appears to be limited to a missing security group. I was able to reproduce this on my end and adding the above rule allowed worker to reach master on port 6443. For the sake of completion, below are a few additional checks you could do:
Make sure both master and worker are in the same VPC and subnet.
If master and worker are in different subnets, make sure appropriate rules are in place to allow both instances talk to each other.

Related

Accessing local dns from local devices using dnsmasq

I got a webserver running on my Mac on localhost:3000 and I am trying to set a local DNS with dnsmasq and to be able to access that DNS from local devices (iPhone / iPad) for test purpose.
I followed this previous post:
iPhone: add entry to /etc/hosts without jailbreaking
I am looking to redirect all *.localhost here
dnsmasq.conf:
/etc/resolver/localhost:
dig google.com:
The Mac Network DNS config:
On my phone, on the same network, adding the Mac Local IP as DNS:
Wi-Fi is connected to xxxxxx and has the IP address 192.168.1.11.
I am able to connect to myapp.localhost:3000 successfully on the Mac but getting Server cannot be found on the iPhone.
Must be missing something there.
EDIT #0: 2021/07/08
netstat -anvp tcp | grep '\b192.168.1.11.53\b' output:
open a terminal and use netstat to check if your dnsmasq is open on 192.168.1.11
if the result is similar to
root#dns-01:~# sudo netstat -tnlp
tcp 0 0 127.0.0.1:53 0.0.0.0:* OUÇA 13376/dnsmasq
it means that it will only accept queries from the local machine, not from your network.
to query from your network you must see something like
tcp 0 0 192.168.1.11:53 0.0.0.0:* OUÇA 13376/dnsmasq
or
tcp 0 0 0.0.0.0:53 0.0.0.0:* OUÇA 104287/dnsmasq

Yet another telnet to 25 on AWS EC2 not working

I went through all these error questions that I could find on SO and I've done everything that it appears I should do. Still, I get connection refused when I telnet to port 25
This is what is asked for in other questions:
netstat-an | grep LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:53045 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 :::111 :::* LISTEN
tcp 0 0 :::80 :::* LISTEN
tcp 0 0 :::22 :::* LISTEN
tcp 0 0 :::443 :::* LISTEN
tcp 0 0 :::52266 :::* LISTEN
So, port 25 is listening. Next, can I telnet to 25 on localhost? Yes. No problem. I can do localhost and I can do it with 127.0.0.1 and I can do it with the proper IP address. As long as I am on the machine itself, I can telnet to port 25.
Next, the EC2 firewall. There are two levels, iptables and the EC2 security zone. I made sure iptables and ip6tables were shut down. Service shows that both are "Firewall is not running". I checked the EC2 security zone. It shows:
25 tcp 0.0.0.0/0
So, it is allowing all traffic to port 25.
Still, I get connection refused when I telnet to port 25.
I continued. I checked /etc/hosts.allow and /etc/hosts.deny. Both are empty.
I continued. I looked in the mail folder. The domain is in local-host-names (which doesn't matter since I never get to the point of entering a recipient email address). I don't see anything about blocking or allowing hosts. Perhaps there is something buried in the cryptic sendmail.cf file. So, I wanted to ensure that sendmail was definitely listening with
lsof -i :25
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sendmail 11457 root 4u IPv4 830292 0t0 TCP *:smtp (LISTEN)
Same as netstat. Listening on all devices on port smtp (25).
So, I am at a loss. Why would I get connection refused on port 25? It isn't unable to reach server. It isn't that it is not listening. It is clearly that it is listening and refusing connections.
The problem here is that there are two issues:
1) Many ISPs block outbound port 25 requests. So, it is rather common that telnet somedomain.com 25 will fail and report something like "denied" or "unreachable."
2) Proper programs that go out on port 25 are mail programs. A mail program is designed to send email to your email service, which then uses the MX record for the domain name to deliver the email. What I found is that when you register a domain name with Network Solutions and set "All" IP addresses to be a certain address, they do NOT set the MX record address. That remains set to Network Solutions' mail server. So, if someone uses a proper email program to hit your domain name, it will go to mx.yourdomain.com, which may not be your server.
In my case, I had to manually set the IP address for the mx record with Network Solutions. Then, I magically started getting emails.
For those that don't know how to check MX records, in Linux, you can run dig domain.name MX. However, it isn't your MX record setting that matters. It is your mail server's setting. For example, if my computer says the mx record is 12.34.56.78 and I use GMail and Google says the mx record is p.ctmail.com, then sending email through GMail will fail.

Connect to server assigned Private IP on Ubuntu Instance

I hope I'm asking this clearly. I connected to a fresh Ubuntu 64-bit Amazon EC2 server with SSH and have set up a Server that uses the IP 172.30.0.72:27015 and it is ONLINE, and my Amazon Security is set to allow all ports.
When I type netstat -atunp | grep srcds_linux it shows:
tcp 0 0 0 172.30.0.72:27015 0.0.0.0:* LISTEN 1277/sccds_linux
udp 0 0 0 172.30.0.72:27015 0.0.0.0:* 1277/sccds_linux
udp 0 0 0 172.30.0.72:26901 0.0.0.0:* 1277/sccds_linux
When I try to access this from outside, I can't. Do I need to be figuring out how to port forward? If so, I know my public IP is 55.174.94.92, how do I connect my client to the server that has the Private IP address? I also know how to make it static with elastic IP's, but haven't done that until I get this to work first.
use the public IP to connect. To access your service at 172.30.0.72:27015 use 55.174.94.92:27015.
If you are no able to do it, then check your security group settings.

Error connecting rabbitmq cluster on Amazon EC2

I am experiencing some difficulties connecting two RabbitMQ nodes on amazon EC2.
The two nodes are controlled using puppet, here is my rabbit.config file:
[
{mnesia, [{dump_log_write_threshold, 1000}]},
{rabbit, [
{tcp_listeners, [5672]},
{kernel, [{inet_dist_listen_min, 55700},{inet_dist_listen_max, 55800}]} ,
{cluster_nodes, ['rabbit#server1', 'rabbit#server2']}
]
}
].
I believe the rights ports for the cluster to connect are open. I am able to telnet from server2 to server1 on both 5672 and 4369.
I have the same /var/lib/rabbitmq/.erlang.cookie on both servers.
And from erlang command line when I net_admin:ping the other node I get pang back.
However, when I run cluster_status on any node they do not look like they are aware of each other. Doing stop_app, reset,rabbitmqctl cluster rabbit#server1 I always get the following error:
Error: {no_running_cluster_nodes...
Has anybody solved a similar problem, or know how to solve it?
Have you opened the ports between 55700 and 55800?
Try checking this to understand what other ports RabbitMQ listens on:
netstat -plten | grep beam
And I'd double-check the cookie...
Like Ivan suggests, you can check which ports the servers are listening on first and then add those TCP rules to Security Groups for servers. That's a good first step.
netstat -plten | grep beam
Returns the following (if server still running and not stop_app)
tcp 0 0 0.0.0.0:37419 0.0.0.0:* LISTEN 498 118739 15519/beam
tcp 0 0 0.0.0.0:15672 0.0.0.0:* LISTEN 498 119032 15519/beam
tcp 0 0 0.0.0.0:55672 0.0.0.0:* LISTEN 498 119029 15519/beam
tcp 0 0 :::5672 :::* LISTEN 498 119018 15519/beam
Notice the common ports 5672 15672 55672 for amqp and web server and the other port is the port the cluster is listening on. Check your other instances and make sure your range includes both of them, then retry and it will work.
Security Group > Inbound > TCP Rule:
30000-65535 and the Security Group allowed sg-XXXXXX and repeat for reciprocating security groups and don't forget to "Apply Rules".
Next make sure you share the /var/lib/rabbitmq/.erlang.cookie (just copy from one server to all others and restart instances)
Then on your command line:
[root#ip-172-31-27-150 ~]# rabbitmqctl stop_app
Stopping node 'rabbit#ip-172-31-27-150' ...
...done.
[root#ip-172-31-27-150 ~]# rabbitmqctl reset
Resetting node 'rabbit#ip-172-31-27-150' ...
...done.
[root#ip-172-31-27-150 ~]# rabbitmqctl join_cluster rabbit#ip-172-31-28-79
Clustering node 'rabbit#ip-172-31-27-150' with 'rabbit#ip-172-31-28-79' ...
...done.
Lastly, don't forget to restart your instance rabbitmqctl start_app
This worked for me on 5 EC2 instances.
thanks for your answer, what I did is to remove the content of this directory except .erlang.cookie ( rm -R /var/lib/rabbitmq/ ). And the cluster connected successfully.
Cheers!

Binding memcached and beanstalk to external IP on an EC2 instance

I had a running EC2 server with an elastic IP address that was happily running memcached and beanstalkd listening on the external IP at port 11211 and 11300 respectively. I took a AMI of the server to upgrade the size and when the server restarted, neither memcached nor beanstalkd were running. Troubling for memcached as this was init.d to restart but beanstalkd was run as daemon from command line.
However, after several hours, I can't seem to figure out how to get them up and running again.
For memcached the config file reads:
# Default connection port is 11211
-p 11211
# Specify which IP address to listen on. The default is to listen on all IP addresses
-l <EXTERNAL IP ADDRESS>
When I try and run memcached as root from the command line:
memcached -d -m 1024 -u root -l <EXTERNAL IP ADDRESS> -v -p 11211
I get:
bind(): Cannot assign requested address
failed to listen on TCP port 11211: Cannot assign requested address
Yet when I look at netstat, nothing is bound to that port already:
netstat -ant | grep LIST
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN
tcp6 0 0 :::22 :::* LISTEN
And if I grep the processlist, there is no other memcached running.
Iptables reports the following:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp spt:11300 state ESTABLISHED
ACCEPT tcp -- anywhere anywhere tcp spt:11211 state ESTABLISHED
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
The EC2 instance is using the same security group as the original server, so the required ports are open.
Finally, maybe a hint is that the ifconfig IP address is different to the elastic IP address. If I try listening on either the inet adds: Bcast: the memcached service runs and binds however I can't seem to access memcached via the elastic IP.
I MUST be doing something stupid and obvious to you. Thanks for your help.
bind to 0.0.0.0 that should be public.

Resources