Lets say Kafka is running as a single node broker on an AWS EC2 instance. The instance has the internal private IP 10.0.0.1. I want to connect to that broker directly from the same EC2 instance and from another EC2 instance in the same VPC and subnet. The security groups are allowing the connection.
Which settings do I have to use to get the connection running?
I tried listeners=PLAINTEXT://0.0.0.0:9092 and advertised.listeners=PLAINTEXT://0.0.0.0:9092. With that setting I can connect to the broker from local (the same instance where the broker is running), but I can't reach the broker from the second EC2 instance.
Does anybody have any idea?
If you are trying to connect to the Kafka instance inside of AWS from one EC2 instance to another the internal ip address should work.
The producer and consumers should make use of the internal private ip addresses as well for both the broker and zookeeper.
Additionally, you may need to verify that the IP Tables at the OS level aren't blocking the communication.
Related
I have a single Kafka broker running in my local mac port 9092 and a topic produced in my local mac as well. I wanted to run a consumer in EC2 to consume my local mac broker topic. I have enabled inbound and outbound access to TCP port 9092 for all IP (0.0.0.0/0) in the EC2 security group.
When I run a consumer command in EC2:
bin/kafka-console-consumer.sh --bootstrap-server <localmac IP>:9092 --topic <topicname> --from-beginning
I get connect timeout error(org.apache.kafka.common.errors.TimeoutException:).
What other outbound/inbound rules in EC2 security group Am I missing here for EC2 to access the local mac address.
You need to configure an advertised listener on your broker for the EC2 consumer to connect to. At the moment you're connecting to the broker on the remote IP of your machine but the broker returns the local IP to the consumer for it to continue its requests to.
Ref: https://rmoff.net/2018/08/02/kafka-listeners-explained/
I have a spring MVC application and I am connecting it to MongoDB cluster
This is in the application.properties file
mongodb.url=mongodb://userName:Password#xx.xx.x.xx:27017,xx.xx.x.xx:27017,xx.xx.x.xx:27017/?authSource=admin
The cluster is deployed on GCP with one primary and 2 secondary servers.
However, after deployment when I hit the API to get the data I get an error
{java.net.UnknownHostException: mongodb-3-arbiters-vm-0}}, {address=mongodb-3-servers-vm-1:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-3-servers-vm-1}, caused by {java.net.UnknownHostException: mongodb-3-servers-vm-1}}
The external IPs are getting mapped to the server name on the GCP dashboard. xx.xx.xx.xx:27017 to mongodb-3-servers-vm-1:27017, hence resulting in unknown host exception. what to do to avoid that ?
When connecting to a replica set, the hostnames, IP addresses and port numbers provided in the connection string are the seedlist.
The driver will connect to the hosts in the seedlist in order to get an initial connection. It uses this connection to perform server discovery. It queries the server that is connected first for the host names, port numbers, and status of the other members of the replica set. The server obtains this information from the replica set configuration document.
This means that the hostnames and port number you used when running rs.initiate or rs.add must be resolvable by both the replica set members and each client host that will be connecting.
There is a feature that supports passing remote clients a different host name, similar to split-horizon DNS, but outside of the git repository, I don't see any mention of it.
We have kafka and zookeeper installed on a single AWS EC2 instance. We have kafka producers and consumers running on separate ec2 instances which are on the same VPC and have the same security group as that of kafka instance. In the producer or consumer config we are using the internal IP address of the kafka server to connect to it.
But we have noticed that we need to mention the public IP address of the EC2 server as advertised.listeners for letting the producers and consumers connect to the Kafka server:
advertised.listeners=PLAINTEXT://PUBLIC_IP:9092
Also we have to whitelist the public ip addresses and open traffic on 9092 port of each of our ec2 servers running producers and consumers.
We want the traffic to flow using internal IP addresses. Is there a way we need not whitelist the public ip addresses and open traffic on 9092 port for each one of our servers running producer or consumer?
If you don't want to open access to all for either one of your servers, I would recommend adding a proper high performance web server like nginx or Apache HTTPD in front of your applications' servers acting as a reverse proxy. This way you could also add SSL encryption and your server stays on a private network while only the web server would be exposed. It’s very easy and you can find many tutorials on how to set it up. Like this one: http://webapp.org.ua/sysadmin/setting-up-nginx-ssl-reverse-proxy-for-tomcat/
Because of the variable nature of the ecosystem that kafka may need to work in, it only makes sense that you are explicit in declaring the locations which kafka can use. The only way to guarantee that external parts of any system can be reached via an ip address is to ensure that you are using external ip addresses.
I have an EC2 instance (running kafka) which needs to access itself via public IPs, but I would like to not open the network ACLs to the whole world.
The rationale is that when a connection is made to a kafka broker, the broker advertises which kafka nodes are available. As kafka will be used inside and outside EC2, the only common option is for the broker to advertise its public IP.
My setup:
an instance, with public IP (not an elastic IP)
a vpc
a security group, allowing access to the kafka ports from my work network
an internet gateway
a route allowing external access via the gateway
The security group is as follow:
Custom TCP Rule, proto=TCP, port=9092, src=<my office network>
Custom TCP Rule, prtot=TCP, port=2181, src=<my office network>
In short, all works fine inside the instance if I use localhost.
All works fine outside the instance if I use the public IP.
What I now want is to use kafka from inside the instance with the public IP.
If I open the kafka ports to the whole world:
Custom TCP Rule, proto=TCP, port=9092, src=0.0.0.0/0
Custom TCP Rule, prtot=TCP, port=2181, src=0.0.0.0/0
It works, as expected, but it does not feel safe.
How could I setup the network ACL to accept inbound traffic from my local instance/subnet/vpv (does not matter which) without opening too much?
Well, this is not clean, but it has the added advantage of not having to pay for external bandwidth.
I did not find a way as I expected (via the security groups), but just by updating the /etc/hosts on my ec2 instance, and actually using a hostname instead of an IP, all works as expected.
For instance, if I give the instance the hostname kafka.example.com, then by having the following line in /etc/hosts:
127.0.0.1 kafka.example.com
I can use the name kafka.example.com everywhere, even if it actually points to a different IP depending on where the call is made.
I am trying to run a socket server on an Amazon Web Services EC2 instance. The socket is able to run just fine on it's own, and telnetting locally on the instance can connect, but trying to telnet to the socket from the outside is failing. I have gone into the security groups to ensure the ports I am using are open for both TCP and UDP (though the socket server has been configured for TCP). Is there something else I am missing?
The server might be listening on the loopback interface or ipv6 by default. You can check that by running netstat --listen -p which will show you which program listens on which address/port. How to make the program listen on the external ipv4 interface depends on the program/programming language.