I am running kafka on ec2 instance. So amazon ec2 instance has two ips one is internal ip and second one is for external use.
I created producer from local machine, but it redirect to internal ip and give me connection unsuccessful error. Can anybody help me to configure kafka on ec2 instance, so that I can run producer from local machine. I am tried many combinations but didn't work.
In the Kafka FAQ (updated for new properties) you can read:
When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in bootstrap.servers in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip and port to be registered in ZK by setting the advertised.listeners property in server.properties.
I solved this problem, by setting advertised.host.name in server.properties and metadata.broker.list in producer.properties to public IP address and host.name to 0.0.0.0.
The easiest way how to reach your Kafka server (version kafka_2.11-1.0.0) on EC2 from consumer in external network is to change the properties file
kafka_2.11-1.0.0/config/server.properties
And modify the following line
listeners=PLAINTEXT://ec2-XXX-XXX-XXX-XXX.eu-central-1.compute.amazonaws.com:9092
Using your public address
Verified on 2.11-2.0.0
I just did this in AWS. First get the Kafka server to listen on the correct interface/IP using host.name. For your case this would be the internal IP, not localhost, since your intent is for outside Kafka clients to connect. Any local clients will need to use that same address, not localhost.
Then set advertised.host.name to a host name, not an IP address. The trick is to get that host name to always resolve to the correct IP for both internal and external machines. I use /etc/hosts inside and DNS outside. See my full answer about Kafka and name resolution here.
If you want to access from LAN, change following 2 files-
In config/server.properties:
advertised.listeners=PLAINTEXT://server.ip.in.lan:9092
In config/producer.properties:
bootstrap.servers=server.ip.in.lan:9092
In my case, the server.ip.in.lan value was 192.168.15.150
Below are the steps to connect Kafka from outside of EC2 instance.
Open Kafka server properties file on EC2.
/kafka_2.11-2.0.0/config/server.properties
Set the value of advertised.listeners to
advertised.listeners=PLAINTEXT://ec2-xx-xxx-xxx-xx.compute-1.amazonaws.com:9092
This should be your Public DNS (IPv4) of EC2 instance.
Stop Kafka server.
Start Kafka server to see above configuration changes in action.
Now you can connect to your Kafka of EC2 instance from outside or from your localhost.
Tried and tested on kafka_2.11-2.0.0
SSH to your EC2 instance or wheverver you're hosting Kafka.
sudo nano /etc/hosts
Add:
127.0.0.1 <your-host-name> localhost
In my case it's:
127.0.0.1 ec2-12-34-56-78.ap-southeast-1.compute.amazonaws.com
Save and exit.
For EC2 you should edit the /etc/hosts file to add:
XXX.XXX.XXX.XXX ip-YYY-YYY-YYY-YYY
where XXX... is your external IP and the ip-YYY-YYY-YYY-YYY is the string returned by the hostname command. You can use 127.0.0.1 instead of your external IP to communicate inside the server.
host.name is deprecated - as are advertised.host.name and advertised.port
Related
I have a 4 node hadoop cluster on ec2. We have configured Hortonworks Hadoop (HDP version 2.4) through Ambari.
I have opened all traffic for our all four instances internally and the office external IP.
Whenever I do telnet within the cluster using internal IP:
telnet <internal_ip> 2181
It is able to connect to the specific port I have my service(zookeeper) running on.
When I use public IP of the same instance(Elastic IP) instead of internal IP, I am not able to telnet either within the cluster or from my office IP:
telnet <elastic_ip> 2181
I have already configured security group to allow all traffic. I am using Ubuntu 14.04. We are not using any other firewall except AWS security group.
Please suggest how can I connect using Elastic IP/Public IP of my instance on this port.
Please find the screenshot of Security Group of EC2:
Do you use the default VPC ?
If not, check if the VPC has an Internet Gateway, the Route table (you need a route to the Internet Gateway) and the Networks ACLs.
The Route table and Network ACLs are applied to a subnet.
The default VPC is configured to allow outside traffic, not a new VPC.
Or, the Elastic IP is linked to the same network interface ? The Elastic IP is linked to a network interface of an instance.
EDIT: you can take a look on AWS doc for a better explanation:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html
I have setup ratchet on amazon EC2 instance.
It is running fine, but I am not able to connect with the public IP address of the instance as well as the url.
Following is the screenshot showing ratchet is running.
You need to make sure that the service is running on the right IP address (not only on 127.0.0.1) and that you enabled access in the security group to that particular port.
I’m trying to set up a clustered deployment of RabbitMQ in a VPC in EC2 based on the documentation here (https://www.rabbitmq.com/clustering.html) and here (https://www.rabbitmq.com/ec2.html)
We currently don't have Route 53 set up within our VPC and rely on the private IP addresses for connections between instances.
I've been trying to get the cluster working without setting up Route 53 by using the private IP address as the hostname as follows:
(Assuming private IP address is 10.0.1.33)
Alter /etc/hostname changing
ip-10-0-1-33
to
10.0.1.33
Alter /etc/hosts changing
127.0.0.1 localhost
to
127.0.0.1 localhost 10.0.1.33
Change hostname ie sudo hostname 10.0.1.33 (or reboot instance to pick up new hostname)
Add file rabbitmq-env.conf to /etc/rabbitmq with contents:
USE_LONGNAME=true
This seems to work and allow me to cluster the rabbit nodes using rabbitmqctl since rabbit treats the private IP address as a fully qualified domain name (USE_LONGNAME is necessary or else rabbit just uses the part of the IP address before the first dot) and the IP addresses are resolvable in the VPC - I get the nodes named rabbit#10.0.1.33 etc
My question is - is there anything I'm missing here or is this a reasonable approach?
I've spent some time now looking for information regarding elasticsearch.yml configurations that make my single instance Elasticsearch (on Windows 2012 Server EC2) accessible via public ip, but everytime I uncomment one or both of following settings the only thing that changes is, calling the private ip as well results in an error.
network.publish_host: <public ip>
network.bind_host: <private ip>
Is this correct and are there any other settings that have to be defined? Shouldn't it run with the default values?
This is more of a general answer as to how networking works within EC2 instead of a specific answer to your question. But it should help inform how to configure your application.
EC2 has 1:1 NAT between a public and private IP address. Because of this, only the private IP address is visible to the instance directly.
If you are binding a service to a network interface, it would be the one with the private IP.
Some services do require knowledge of the external IP address in order to function properly. The only one I have run into is FTP in a passive configuration, likely due to the fact that it needs to open a separate socket for data transfer.
In the case of elastic search, it appears that they have a special plugin that will help configure elastic search for the aws environment: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-network.html
I had the same problem.
Installed only one instance of ES on aws EC2 and wanted to grant it public access.
On ubuntu 16.04 this is what works for me:
in /etc/elasticsearch/elasticsearch.yml add this line:
network.host: <ec2 instance private ip>
The private ip should be something like 172.x.x.x
Also do not forget allow access in security group in your aws console for port 9200 (default) and ip address from which you will be sending requests.
So difference was setting not public but private ip address from aws console..
Also note that this can be dangerous as there is not any user/password or other access control
I am having trouble connecting to an Amazon Elastic Cloud Computer Instance via a browser.
I attempted going to ********.compute-1.amazonaws.com , but the browser returns that the connection has timed out.
I can connect via ssh and winscp. That is how I uploaded a web app I developer. I have also created a security group and added rules to open ports 22 and 80.
Do I have to assign the security group to the instance somehow?
The security group's rules also do not have a source IP, well they do its 0.0.0.0/0
I would really appreciate any and all help in getting this site ' viewable ' via a browser.
By default, your instances will only be in the default security group. If it's an EC2 instance you cannot change security groups while the instance is running, you'll have to specify them in advance. If it's a VPC instance you can change security groups at runtime.
Add the rule to the default group
You can however add the rule to allow port 80 to that default security group; just don't create a new security group as it can not be associated with the running instance.
Is the web server up?
Also, make sure that your web server is up and running. From your instance (using SSH shell access), check if the right process is listening on port 80, using the command netstat -lnp. You should then see a row with proto tcp and a Local Address ending in :80. The IP Address listed should be either 0.0.0.0 (meaning 'any IP') or a specific IP of a listening network interface.
Web server not up
If you are in need of a web server, take a look at Apache or Nginx. They both support PHP.
Hope this helps.
I had also faced similar issue with ec2 micro instance. I was using Red-Hat AMI. Despite of opening ports 8081 in security group, I was not able to a telnet to the host port. Disabling the iptable did the trick for me:
sudo /etc/init.d/iptables stop
Do not forget to disable firewall if you use windows for your server.
I faced the same issue while setting up redash AMI image on AWS. Inbound security rules should be changed when instance is not running. Let's say if the instance is running (meaning it's active and started); If you change the inbound rules of that machine you'll still face firewall issue. So Stop the machine on which you want to change the inbound rules on. Change the inbound rules. Start the machine now. Now you can hit the machine url from the ip you just opened the access to the machine to.
The EC2 instance firewall is maybe enabled.
Check it with this command:
sudo systemctl status firewalld
if enabled you can disable it with :
sudo systemctl disable firewalld
or setup rules to allow port 80 trafic