I have a .NET Kafka client (using librdkafka via a Confluent's .NET client) running on a physical server with two network interfaces active. One is 10G and the other is 1G, both of them have static IP addresses assigned. Our networking team handles the configurations and is unlikely to change their practices for one application so I'd like to handle this client-side. I should also mention that the 1G interface and 10G interfaces are on the same network.
Since my Kafka cluster (3-node) is all 10G, I would like to require my application's consumer to bind to the 10G IP address. Looking through all of the documentation, I can't find anything about defining this on the client.
I would like to avoid any "hacky" solutions like setting Kafka to deny any non-whitelisted IP addresses or DNS tomfoolery.
Thanks in advance!
Just to be sure.., Do you know if your server is doing interface bonding (means the traffic will load balance between each interface, though, it's unlikely to do binding on different speed interfaces..)?
If not, as your two interfaces are on the same network, it means you will only use one interface to reach the network (except if you have exotic routing config). This interface will be defined by your default route.
If it's a Linux server, you can do as follows :
ip route
default via X.X.X.X dev YOURDEFAULTINTERFACE
If it's the 10G, you have nothing to do and you can be sure it will use this interface.
If not, you cannot do anything Kafka side, as it's purely OS settings side. Your Kernel will forward any traffic through this default interface.
Again.. I insist on the fact that this is because both your interfaces are in the same network.
If you have any doubts with this, please share your network configuration in details ( result of ip addr and ip route)
Yannick
Related
I was wondering if typing remote server url instead of typing localhost in spring boot properties db url (spring.datasource.url) is slower? Let's say I am running spring boot application on server with IPv4 123.123.12.12, will typing
jdbc:mariadb://123.123.12.12:3306/dbname
make it slower than
jdbc:mariadb://localhost:3306/dbname ?
When you access localhost, your /etc/hosts file will tell your computer not to look any further and redirects you to your own computer. When you access the IP address, your computer will ask the router to fetch the data, and your router will then point back to your computer.
Directly using the IP address of any interface on the localhost - either the loopback interface (127.0.0.1) or any other - is the option with the absolutely best performance. The packets will be actually routed through the loopback interface (no matter which IP is actually used) at - practically - CPU speed.
There are three reasons, however, to prefer 127.0.0.1 over the IPs of the other interfaces:
The loopback interface is crucial to the operation of the system and
as such it is initialized very early in the boot process and nearly
always available.
It is not affected by external factors: while removing the eth0 cable
will not by itself interrupt localhost's access to itself via eth0's
IP, it will mess things up if you have any of the many
"autoconfiguration" systems that will happily shutdown the interface
on link loss.
If you have a firewall setup, it's quite possible that the rule chain
is longer (and thus slightly worse performance-wise) when the IPs of
the public interfaces are involved.
Loopback, Also Refer This
Yes, setting up IP or DNS are slower than localhost. In case of localhost, application does not need to verify anything. It will directly try to connect the database on same server. But in case of IP and DNS, it first need to check that provided url is valid and then will connect to database.
I have an EC2 instance (running kafka) which needs to access itself via public IPs, but I would like to not open the network ACLs to the whole world.
The rationale is that when a connection is made to a kafka broker, the broker advertises which kafka nodes are available. As kafka will be used inside and outside EC2, the only common option is for the broker to advertise its public IP.
My setup:
an instance, with public IP (not an elastic IP)
a vpc
a security group, allowing access to the kafka ports from my work network
an internet gateway
a route allowing external access via the gateway
The security group is as follow:
Custom TCP Rule, proto=TCP, port=9092, src=<my office network>
Custom TCP Rule, prtot=TCP, port=2181, src=<my office network>
In short, all works fine inside the instance if I use localhost.
All works fine outside the instance if I use the public IP.
What I now want is to use kafka from inside the instance with the public IP.
If I open the kafka ports to the whole world:
Custom TCP Rule, proto=TCP, port=9092, src=0.0.0.0/0
Custom TCP Rule, prtot=TCP, port=2181, src=0.0.0.0/0
It works, as expected, but it does not feel safe.
How could I setup the network ACL to accept inbound traffic from my local instance/subnet/vpv (does not matter which) without opening too much?
Well, this is not clean, but it has the added advantage of not having to pay for external bandwidth.
I did not find a way as I expected (via the security groups), but just by updating the /etc/hosts on my ec2 instance, and actually using a hostname instead of an IP, all works as expected.
For instance, if I give the instance the hostname kafka.example.com, then by having the following line in /etc/hosts:
127.0.0.1 kafka.example.com
I can use the name kafka.example.com everywhere, even if it actually points to a different IP depending on where the call is made.
I am working on some legacy code on Windows for a desktop app in "C.
The client needs to know the geo-location of the user who is running the application.
I have the geo-location code all working (using MaxMind: http://dev.maxmind.com/).
But now I'm looking for help in getting their external IP.
From all the discussions on this topic throughout SO and elsewhere it seems that there is a way to do this by connecting to a "reliable" host (server) and then doing some kind of lookup. I'm not too savvy on WinSock but this is the technology that may be the simplest to use.
Another option is to use WinHttpConnect technology.
Both have "C" interfaces.
Thank you for your support and suggestions.
You can write a simple web service that checks the IP address(es) that the program presents when connecting to that web service.
Look at http://whatismyip.com for an example.
Note that multiple addresses can be presented by the HTTP protocol if there are proxy servers along the route.
You can design your simple web service to get the IP of the client. See
How do I get the caller's IP address in a WebMethod?
and then return that address back to the caller.
Note that in about 15% of cases (my experience metric) the geo location will be way off. The classic example is that most AOL users are routed through a small number of proxy servers. However, there are many other cases where the public IP does not match the user's actual location. Additionally, Geo IP databases are sometimes just wrong.
Edit
It is not possible to detect your external IP address using only in-browser code.
The WebSocket has no provision to expose your external IP address.
https://www.rfc-editor.org/rfc/rfc6455
You need an outside server to tell you what IP it sees.
One of my real weak points in programming is networking, so I admit that I may be a little over my head with this project. Please feel free to tell me if what I'm trying to do doesn't make any sense
What I am trying to do, basically, is run a program on my laptop (Node.JS, probably) that handles requests from a website, does some functions, and serves data back to a client running on the website. (Research tells me this is called an RPC server)
When you listen for requests in Node.JS, you specify a port and optionally an IP Address- localhost, 127.0.0.1, is what all the tutorials I've read have used, but that's not sufficient for what I'm trying to do
I've read that I'll need to set up a static IP Address? But I think those are relative to my LAN, so they'll be like 192.168.0.X. So then what would I specify for the IP for the server and the client? (I don't think the port particularly matters). Do I need a DNS?
I hope this makes sense, sorry for so many questions, thank you for your help
You can run a server on your local machine, and you will specify your local IP address for the script, like 192.168.0.x. But for this server to ever receive a connection, your client must connect to your external IP address. It is the IP address that you get from your Internet provider when you connect to Internet. If your external IP is static, i.e. it does not change, then you can use it in your client script. If the external IP changes, you must setup a DNS record that would resolve the name of your computer. DynDNS can be used for that purpose.
If you have a router, it must be setup so that it forwards connections to your laptop where the server runs. And your firewall must be configured to allow connections.
I wrote a tool running on a system (Win7) with two network interfaces, each linked to a different subnet, each with its own gateway which is then linked to two separate distant networks (there are outgoing firewalls after each gateway). I’m initiating outgoing TCP connections via both NICs by using Socket.Bind (before doing Connect) to each relevant NIC’s IP address. First NIC is working fine, but for the second NIC, I’m getting SocketException: “A socket operation was attempted to an unreachable network”.
My original understanding was that since sockets are bound to concrete NIC’s local endpoint, which has its gateway defined, the connection should be routed to this gateway and therefore should work. However, it seems that source IP address is ignored and the routing is working according to local routing table (i.e. second NIC’s connect request goes to first, default, network and being rejected because it has wrong subnet).
Adjusting local routing tables helps, but it makes me wonder about the whole reasoning behind ability of the socket to bind to specific local IP.
Doing some extra reading, I found out that, indeed, there’s such thing as “source IP routing”, but it is disabled in Windows by default (via DisableIPSourceRouting registry setting), due to security reasons, as described, e.g. here:
http://msdn.microsoft.com/en-us/library/ff648853.aspx
http://www.bloggersbase.com/disableipsourcerouting/
Questions:
If my original understanding was correct (i.e. Socket.Bind should be enough) – why it is not working without modifying routing tables?
If my understand was NOT correct (i.e. Socket.Bind is ignored and routing is used) – what’s the point of having Socket.Bind? Why doing it at all?
Also, I’d like to understand better, what is the actual risk of having source IP routing enabled (preferably with example of a possible exploit)?
Any ideas of solving the requirement without manually modifying local routing table will be greatly appreciated.
Many thanks.
OK, after some reading, here are some high-level explanations on what's happening. I still need to verify the below conclusions in my system. Apparently, local binding is typically ignored when selecting network interface. Instead, routing table is used for this. However, in Strong Host Model (default for Vista and newer, non-existant in XP), source IP is used as a 'constraint' in the routing table lookup.
Brief explanation about strong host model vs. weak host model:
http://technet.microsoft.com/en-us/magazine/2007.09.cableguy.aspx
Explanation on what's different in XP vs newer Windows versions in respect to the above:
http://blogs.technet.com/b/networking/archive/2009/04/24/source-ip-address-selection-on-a-multi-homed-windows-computer.aspx