I was starting with the Google Cloud Platform's microservice demo. And I was curious how gRPC stubs work when the services are deployed in containers.
As far as my understanding goes, the container of a particular service are addressed by the Service IP specified in the YAML configuration file. So the gRPC server of a service must listen at the service IP? But I came across the following snippet of code:
l, err := net.Listen("tcp", fmt.Sprintf(":%s", port))
if err != nil {
log.Fatal(err)
}
I am wondering how does the server listen to an address without an IP?
:{port} isn't an "address without an IP".
The documentation for Listen includes "if the host in the address parameter is empty or a literal unspecified IP address, Listen listens on all available unicast and anycast IP addresses of the local system".
So, in this case, without a host address, the effective address would be 0.0.0.0 which corresponds to all interfaces. Corollary a common mistake people make when using containers is to bind their code to localhost (127.0.0.1) which cannot be accessed from outside the container.
Using 0.0.0.0 is a common (good) practice, particularly when using containers, as it effectively delegates address binding to the container runtime.
So, your app runs on {port} on all interfaces within the container. The container runtime then binds (one of more of) these interfaces to the host's interface(s) and your e.g. client code connects to the host's IP address.
When your container is being managed by Kubernetes, Kubernetes assigns IP address(es) to the containers running your app and these are often exposed to other services using a Kubernetes Service resource which not only has an IP address but also a cluster DNS.
The Kubernetes YAML probably specifies a Service DNS.
Kubernetes resolves requests to the DNS name to a selected container (IP and port)
The container runtime routes incoming requests on the host's port to the container's port
Your gRPC server will accept traffic from the container runtime on any interface that's on the {port} that you've defined it to net.Listen on.
Related
I have a basic web application running on a port in an EC2 instance. I have also created an A record in Route53 which points a domain name that I own towards the ip address of the EC2 instance.
When I directly access ip-address:port, it works fine, but because the a record does not point towards that specific port, the website cannot be accessed through the domain name.
How do I specify a port that all requests to that ip address should be routed to?
port is referenced by protocol you are using.
if you use http you will connect to port 80. https uses 443, ftp 21, smtp 25 and there are many others (they are called default ports for service but they may use every other defined).
You are probably running application, listening on some different port (but using protocol http) and that's why you can reach it by entering http://ip-address:port
what you can do to reach it by entering only http://ip-address is to set it to listen on port 80 (you will need root privileges to do this) or set redirect from port 80 to your applications port (you can use pure port redirection using iptables or (better) reverse proxy software. It can be apache as mentioned in comment above, or nginx or haproxy or something else (you haven't specified operating system anyway - those are mainly for linux).
Hope that helps
We have kafka and zookeeper installed on a single AWS EC2 instance. We have kafka producers and consumers running on separate ec2 instances which are on the same VPC and have the same security group as that of kafka instance. In the producer or consumer config we are using the internal IP address of the kafka server to connect to it.
But we have noticed that we need to mention the public IP address of the EC2 server as advertised.listeners for letting the producers and consumers connect to the Kafka server:
advertised.listeners=PLAINTEXT://PUBLIC_IP:9092
Also we have to whitelist the public ip addresses and open traffic on 9092 port of each of our ec2 servers running producers and consumers.
We want the traffic to flow using internal IP addresses. Is there a way we need not whitelist the public ip addresses and open traffic on 9092 port for each one of our servers running producer or consumer?
If you don't want to open access to all for either one of your servers, I would recommend adding a proper high performance web server like nginx or Apache HTTPD in front of your applications' servers acting as a reverse proxy. This way you could also add SSL encryption and your server stays on a private network while only the web server would be exposed. It’s very easy and you can find many tutorials on how to set it up. Like this one: http://webapp.org.ua/sysadmin/setting-up-nginx-ssl-reverse-proxy-for-tomcat/
Because of the variable nature of the ecosystem that kafka may need to work in, it only makes sense that you are explicit in declaring the locations which kafka can use. The only way to guarantee that external parts of any system can be reached via an ip address is to ensure that you are using external ip addresses.
We did a network traffic capture while using the Discovery Node API and found that there was port access attempted on 621XX ports (62111, 62112, etc) and we were wondering if there was a specific set of ports the Discovery service typically uses.
This information would help immensely when firewall and corporate proxy settings come in to play.
The Watson Discovery API is an HTTPS service so only needs TCP port 443 to work. I would suggest that the activity on ports 621** are dynamic or private ports that your app is using to make the connections. They are not ports that need to be punched through firewalls, they are merely the port at which HTTPS connections to the remote server on port 443 are terminated.
I have an EC2 instance (running kafka) which needs to access itself via public IPs, but I would like to not open the network ACLs to the whole world.
The rationale is that when a connection is made to a kafka broker, the broker advertises which kafka nodes are available. As kafka will be used inside and outside EC2, the only common option is for the broker to advertise its public IP.
My setup:
an instance, with public IP (not an elastic IP)
a vpc
a security group, allowing access to the kafka ports from my work network
an internet gateway
a route allowing external access via the gateway
The security group is as follow:
Custom TCP Rule, proto=TCP, port=9092, src=<my office network>
Custom TCP Rule, prtot=TCP, port=2181, src=<my office network>
In short, all works fine inside the instance if I use localhost.
All works fine outside the instance if I use the public IP.
What I now want is to use kafka from inside the instance with the public IP.
If I open the kafka ports to the whole world:
Custom TCP Rule, proto=TCP, port=9092, src=0.0.0.0/0
Custom TCP Rule, prtot=TCP, port=2181, src=0.0.0.0/0
It works, as expected, but it does not feel safe.
How could I setup the network ACL to accept inbound traffic from my local instance/subnet/vpv (does not matter which) without opening too much?
Well, this is not clean, but it has the added advantage of not having to pay for external bandwidth.
I did not find a way as I expected (via the security groups), but just by updating the /etc/hosts on my ec2 instance, and actually using a hostname instead of an IP, all works as expected.
For instance, if I give the instance the hostname kafka.example.com, then by having the following line in /etc/hosts:
127.0.0.1 kafka.example.com
I can use the name kafka.example.com everywhere, even if it actually points to a different IP depending on where the call is made.
Imagine a Windows box, which:
hosts a WCF service
has multiple NICs
sits behind NAT
When a user issues a request to the service (on top of the WCF infrastructure), he uses the external address assigned to the target machine by the NAT.
I have to write some piece of code inside the WCF service, which must know which of the several NICs that the machine has was used to actually handle the network traffic. How does this code identify the NIC is less important - it could be its MAC address (the best) or it could be the (internal) IP address of the NIC.
How can I do it?
EDIT1
I will try to supply the question context. There are two agents. Both expose the same WCF service. In addition, one of the agents can be instructed to start probing the network towards the second agent in the following fashion:
Agent A is asked to probe the network to agent B
Agent A negotiates with agent B the UDP port to utilize for the sake of probing using the WCF service exposed by the agent B.
Once negotiation is over, the agent A starts some custom protocol over UDP, where the agent B acts as the server - i.e. it binds to the UDP port negotiated in the previous item.
Binding to a UDP port requires two pieces - the IP address and UDP port, where the IP address can either be a specific IP address or * (to bind to all the IP addresses associated with the machine). The latter option is not good for us - I will omit the reasons. This leaves us the former option - binding to the specific IP address. However, when the agent B is behind NAT, the IP address used to talk to the WCF service is the external IP address assigned to the agent by the NAT. Binding, on the other hand, requires the respective internal IP address - how to get it?
Can you check the OperationContext.Current.Channel.LocalAddress (it's an EndpointAddress) inside a WCF operation?
As a side note, getting the remote address can be done with:
OperationContext context = OperationContext.Current;
MessageProperties prop = context.IncomingMessageProperties;
RemoteEndpointMessageProperty endpoint =
prop[RemoteEndpointMessageProperty.Name] as RemoteEndpointMessageProperty;
string ip = endpoint.Address;
--larsw
To get the MAC use
System.Net.NetworkInformation.NetworkInterface.GetPhysicalAddress();
All Nics:
System.Net.NetworkInformation.NetworkInterface.GetAllNetworkInterfaces();
to find out what is the real listening ip address you can write a code that listen to your port on each address and ping it from an agent emulator to see that the address is valid.
Cheers,
Gilad