I am running rabbitmq servers over Ec2. I am trying to create a cluster, and have the ports: 4369 and 25672 and 5672 open as specified in the rabbitmq docs : https://www.rabbitmq.com/clustering.html
Whenever I start my rabbitmq server:
rabbitmq-server -detached
The server starts on a different port. Output of epmd -names gives:
epmd: up and running on port 4369 with data:
name rabbit at port 50696
Where '50696' changes each time I stop the server and start it again. This is making it impossible for me to cluster my instances without allowing all ports inbound on my aws firewall rules.
Any ideas on what is going on?
Take a look at RABBITMQ_DIST_PORT here: https://www.rabbitmq.com/ec2.html
rabbitmq.config:
[
{kernel,
[
{inet_dist_listen_min, 55555},
{inet_dist_listen_max, 55560}
]}
].
epmd -names
epmd: up and running on port 4369 with data:
name rabbit at port 55555
Related
From my Spring Boot application deployed as a Docker container I need to be able to access the parent node (within a Swarm) by some kind of docker-exposed DNS name so that I can add it to a configuration file and was wondering if there was a DNS name exposed automatically by Docker for this purpose?
Another container that is scheduled on the same host running in "network_mode: host" is running Consul and advertising on port 8500 (real IP is 192.168.1.233).
If I run netstat on the Swarm node (ip 192.168.1.233) I can see it's listening on port 8500:
# netstat -anp | grep 8500
tcp6 0 0 :::8500 :::* LISTEN 25010/docker-proxy
I want to be able to define a connection string as configuration in the Spring Boot app on the swarm node to the local instance of Consul sheduled on the same physical host as the Spring Boot app.
If I refer to "localhost" within the Spring Boot container then there's nothing listening on the 8500 port, proven by running this after shelling into the Spring Boot container (using a sample Consul API call)
# wget http://localhost:8500/v1/status/leader
Connecting to localhost:8500 (127.0.0.1:8500)
wget: can't connect to remote host (127.0.0.1): Connection refused
I've also tried using "host.docker.internal" but that doesn't work:
# ping host.docker.internal
ping: bad address 'host.docker.internal'
The hosts are all Centos 7 hosts, and Docker version 18.09.1, build 4c52b90.
The firewalld service is disabled on all hosts.
I am running an Akka HTTP service on port 8080 in my EC2 instance. It expects web socket connections, but normal HTTP should return 400 bad request.
I use this code to start Akka service:
Http().bindAndHandle(route, "localhost", 8080)
From the remote terminal I have this expected behavior, but not from outside.
$ wget ec2-XX-XXX-XX-XXX.compute-1.amazonaws.com:8080
--2017-10-01 15:27:31-- http://ec2-XX-XXX-XX-XXX.compute-1.amazonaws.com:8080/
Resolving ec2-XX-XXX-XX-XXX.compute-1.amazonaws.com... XX.XXX.XX.XXX
Connecting to ec2-XX-XXX-XX-XXX.compute-1.amazonaws.com|XX.XXX.XX.XXX|:8080... failed: Connection refused.
Here is my inbound rules configuration:
If I do netstat --listen -p the port 8080 does not appear.
Thanks!
I found out the problem. I should be binding the service to the private IP address of the EC2 instance.
I have an EC2 instance which is running with the following security groups:
HTTP - TCP - 80 - 0.0.0.0/0
Custom UDP Rule - UDP - 1194 - 0.0.0.0/0
SSH - TCP - 22 - 0.0.0.0/0
Custom TCP Rule - TCP - 943 - 0.0.0.0/0
HTTPS - TCP - 443 - 0.0.0.0/0
However, when I try to access http://{PUBLIC_IP} or https://{PUBLIC_IP} in the browser, I get a "{IP} refused to connect" error. I'm new to AWS. Am I missing something here? What should I do to debug?
One way to debug this particular class of problem is to use netcat in order to determine where the problem lies.
If you run netcat against port 80 on the public IP address of your instance and just get a hang (no output at all), then most likely your security group isn't allowing traffic through. Here is an example from an EC2 instance that is in a security group that doesn't allow port 80 traffic inbound:
% nc -v 55.35.300.45 80
<just hangs>
Whereas if the security group is changed to allow port 80, but the EC2 instance doesn't have any process listening on port 80, you'll get the following:
% nc -v 55.35.300.45 80
nc: connectx to 52.38.300.43 port 80 (tcp) failed: Connection refused
Given that your browser gave you a similar "connection refused", most likely the problem is that there is no web server running on your instance. You can verify this by ssh'ing into the instance and seeing if you can connect to port 80 there:
ssh ec2-user#55.35.300.45
% nc -v localhost 80
nc: connect to localhost port 80 (tcp) failed: Connection refused
If you get something like the above, you're definitely not running a webserver.
I'm not sure if it's too late to help but I was stuck with a similar issue with my test server
SG Inbound: ssh -> 22
HTTP -> 80
NACL: default allow/deny settings
but still couldn't ping to the server from my browser, then I realize there's nothing running on the server that can serve the request, and I started httpd server (webserver) and it worked.
sudo yum -y install httpd
sudo service httpd start
this way you can test the connectivity if you are playing with SGs and NACLs and of course it's not the only way, just an example if you're figuring your System N/W out.
Have you installed webserver(ngingx/apache) to serve your requests. If so please share your the config files. (So that it will help to troubleshoot)
I think the reason is probably that you did not set up a web server for your EC2 instance, because if you try to access http://{PUBLIC_IP} or https://{PUBLIC_IP}, you need to have a background server to serve the http request as #Niranj Rajasekaran said.
By the way, by simply pinging the {PUBLIC_IP}, you could see if your connection to your EC2 instance is normal or not.
In command prompt or terminal, type
ping {PUBLIC_IP}
In my case, the server was running but available on just 127.0.0.1 so it refused connections from external hosts. To see if this is your situation, you can run
netstat -an | grep <port number>
If it says 127.0.0.1:<port number> instead of 0.0.0.0:<port number>, you have this problem.
Usually there's a flag or an argument in your server code somewhere to set the host to 0.0.0.0:
app.run(host='0.0.0.0') # flask example
However, in my case, I had already set this so I thought that couldn't possibly be the issue, which is how I ended up on this thread, which asks more generally about the problem. Unfortunately, I was using docker, and had set 0.0.0.0 on the container but was mapping that explicitly to 127.0.0.1 on the host in the docker-compose port-mapping:
ports:
- "127.0.0.1:<port number>:<port number>"
Changing that line to remove the host IP specification fixed the problem upon re-deploy:
ports:
- "<port number>:<port number>"
I have my Single node Hadoop installed on Google Compute Engine instance and i want to open port 50070 on that machine to access the hadoop dashboard. i configured in the firewall rule as tcp:50070 in compute engine networks. but still i am unable to access my port outside the network (ie . via internet). I tried nmap for the public ip of my GCE instance and i got a result which has only ssh port got opened all other ports are filtered .
Note: i am using debian 7.5 image
Make sure your daemon is listening on port 50070. If you have more than one networks in you project make sure the port is opened on the right network. You can run the following commands to check the information about your instance and network.
lsof -i
gcutil --project= getinstance
gcutil --project= listnetworks
gcutil --project= listfirewalls
gcutil --project= getfirewall
Check if IP/Port is allowed in iptables or not.
iptables -L
would show you all the records.
To allow port in iptables you can do the following:
sudo iptables -A INPUT -p tcp -m tcp --dport 50070 -j ACCEPT
sudo iptables-save -c
Short answer
In addition to configure the firewall rule at GCE web console make sure that your server is listening at 0.0.0.0 instead of 127.0.0.1
Long answer
In the context of servers, 0.0.0.0 means all IPv4 addresses on the local machine. If a host has two IP addresses, 192.168.1.1 and 10.1.2.1, and a server running on the host listens on 0.0.0.0, it will be reachable at both of those IPs - Source
In contrast 127.0.0.1 is the IP address used to stablish a connection to the same machine used by the user this address is usually referred as the localhost.
It's often used when you want a network-capable application to only serve clients on the same host. A process that is listening on 127.0.0.1 for connections will only receive local connections on that socket. - Source
Hence, if you try to stablish a connection to your server from internet and your server is listening at 127.0.0.1 at your GCE machine, then, from the server point of view a request has never been received and as a consequence Goocle Cloud Firewall will refuse the connection because there is no server listening at the opened port (in your case 50070).
I hope this answer helps to solve your problem. Best regards.
I am experiencing some difficulties connecting two RabbitMQ nodes on amazon EC2.
The two nodes are controlled using puppet, here is my rabbit.config file:
[
{mnesia, [{dump_log_write_threshold, 1000}]},
{rabbit, [
{tcp_listeners, [5672]},
{kernel, [{inet_dist_listen_min, 55700},{inet_dist_listen_max, 55800}]} ,
{cluster_nodes, ['rabbit#server1', 'rabbit#server2']}
]
}
].
I believe the rights ports for the cluster to connect are open. I am able to telnet from server2 to server1 on both 5672 and 4369.
I have the same /var/lib/rabbitmq/.erlang.cookie on both servers.
And from erlang command line when I net_admin:ping the other node I get pang back.
However, when I run cluster_status on any node they do not look like they are aware of each other. Doing stop_app, reset,rabbitmqctl cluster rabbit#server1 I always get the following error:
Error: {no_running_cluster_nodes...
Has anybody solved a similar problem, or know how to solve it?
Have you opened the ports between 55700 and 55800?
Try checking this to understand what other ports RabbitMQ listens on:
netstat -plten | grep beam
And I'd double-check the cookie...
Like Ivan suggests, you can check which ports the servers are listening on first and then add those TCP rules to Security Groups for servers. That's a good first step.
netstat -plten | grep beam
Returns the following (if server still running and not stop_app)
tcp 0 0 0.0.0.0:37419 0.0.0.0:* LISTEN 498 118739 15519/beam
tcp 0 0 0.0.0.0:15672 0.0.0.0:* LISTEN 498 119032 15519/beam
tcp 0 0 0.0.0.0:55672 0.0.0.0:* LISTEN 498 119029 15519/beam
tcp 0 0 :::5672 :::* LISTEN 498 119018 15519/beam
Notice the common ports 5672 15672 55672 for amqp and web server and the other port is the port the cluster is listening on. Check your other instances and make sure your range includes both of them, then retry and it will work.
Security Group > Inbound > TCP Rule:
30000-65535 and the Security Group allowed sg-XXXXXX and repeat for reciprocating security groups and don't forget to "Apply Rules".
Next make sure you share the /var/lib/rabbitmq/.erlang.cookie (just copy from one server to all others and restart instances)
Then on your command line:
[root#ip-172-31-27-150 ~]# rabbitmqctl stop_app
Stopping node 'rabbit#ip-172-31-27-150' ...
...done.
[root#ip-172-31-27-150 ~]# rabbitmqctl reset
Resetting node 'rabbit#ip-172-31-27-150' ...
...done.
[root#ip-172-31-27-150 ~]# rabbitmqctl join_cluster rabbit#ip-172-31-28-79
Clustering node 'rabbit#ip-172-31-27-150' with 'rabbit#ip-172-31-28-79' ...
...done.
Lastly, don't forget to restart your instance rabbitmqctl start_app
This worked for me on 5 EC2 instances.
thanks for your answer, what I did is to remove the content of this directory except .erlang.cookie ( rm -R /var/lib/rabbitmq/ ). And the cluster connected successfully.
Cheers!